diff options
Diffstat (limited to 'doc')
208 files changed, 5555 insertions, 2681 deletions
diff --git a/doc/README.md b/doc/README.md index 4ff1a0582c8..c30bf328003 100644 --- a/doc/README.md +++ b/doc/README.md @@ -6,8 +6,8 @@ - [API](api/README.md) Automate GitLab via a simple and powerful API. - [CI/CD](ci/README.md) GitLab Continuous Integration (CI) and Continuous Delivery (CD) getting started, `.gitlab-ci.yml` options, and examples. - [GitLab as OAuth2 authentication service provider](integration/oauth_provider.md). It allows you to login to other applications from GitLab. -- [Container Registry](container_registry/README.md) Learn how to use GitLab Container Registry. -- [GitLab Basics](gitlab-basics/README.md) Find step by step how to start working on your commandline and on GitLab. +- [Container Registry](user/project/container_registry.md) Learn how to use GitLab Container Registry. +- [GitLab basics](gitlab-basics/README.md) Find step by step how to start working on your commandline and on GitLab. - [Importing to GitLab](workflow/importing/README.md). - [Importing and exporting projects between instances](user/project/settings/import_export.md). - [Markdown](user/markdown.md) GitLab's advanced formatting system. @@ -20,6 +20,7 @@ - [Webhooks](web_hooks/web_hooks.md) Let GitLab notify you when new code has been pushed to your project. - [Workflow](workflow/README.md) Using GitLab functionality and importing projects from GitHub and SVN. - [University](university/README.md) Learn Git and GitLab through videos and courses. +- [Git Attributes](user/project/git_attributes.md) Managing Git attributes using a `.gitattributes` file. ## Administrator documentation @@ -35,7 +36,7 @@ - [Libravatar](customization/libravatar.md) Use Libravatar instead of Gravatar for user avatars. - [Log system](administration/logs.md) Log system. - [Environment Variables](administration/environment_variables.md) to configure GitLab. -- [Operations](operations/README.md) Keeping GitLab up and running. +- [Operations](administration/operations.md) Keeping GitLab up and running. - [Raketasks](raketasks/README.md) Backups, maintenance, automatic webhook setup and the importing of projects. - [Repository checks](administration/repository_checks.md) Periodic Git repository checks. - [Repository storages](administration/repository_storages.md) Manage the paths used to store repositories. @@ -43,12 +44,13 @@ - [System hooks](system_hooks/system_hooks.md) Notifications when users, projects and keys are changed. - [Update](update/README.md) Update guides to upgrade your installation. - [Welcome message](customization/welcome_message.md) Add a custom welcome message to the sign-in page. -- [Reply by email](incoming_email/README.md) Allow users to comment on issues and merge requests by replying to notification emails. +- [Reply by email](administration/reply_by_email.md) Allow users to comment on issues and merge requests by replying to notification emails. - [Migrate GitLab CI to CE/EE](migrate_ci_to_ce/README.md) Follow this guide to migrate your existing GitLab CI data to GitLab CE/EE. - [Git LFS configuration](workflow/lfs/lfs_administration.md) - [Housekeeping](administration/housekeeping.md) Keep your Git repository tidy and fast. -- [GitLab Performance Monitoring](monitoring/performance/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics. -- [Monitoring uptime](monitoring/health_check.md) Check the server status using the health check endpoint. +- [GitLab Performance Monitoring](administration/monitoring/performance/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics. +- [Request Profiling](administration/monitoring/performance/request_profiling.md) Get a detailed profile on slow requests. +- [Monitoring uptime](user/admin_area/monitoring/health_check.md) Check the server status using the health check endpoint. - [Debugging Tips](administration/troubleshooting/debug.md) Tips to debug problems when things go wrong - [Sidekiq Troubleshooting](administration/troubleshooting/sidekiq.md) Debug when Sidekiq appears hung and is not processing jobs. - [High Availability](administration/high_availability/README.md) Configure multiple servers for scaling or high availability. diff --git a/doc/administration/container_registry.md b/doc/administration/container_registry.md index c5611e2a121..d7cfb464f74 100644 --- a/doc/administration/container_registry.md +++ b/doc/administration/container_registry.md @@ -1,42 +1,32 @@ -# GitLab Container Registry Administration +# GitLab Container Registry administration > [Introduced][ce-4040] in GitLab 8.8. -With the Docker Container Registry integrated into GitLab, every project can -have its own space to store its Docker images. - -You can read more about Docker Registry at https://docs.docker.com/registry/introduction/. - --- -<!-- START doctoc generated TOC please keep comment here to allow auto update --> -<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> -**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* +> **Notes:** +- Container Registry manifest `v1` support was added in GitLab 8.9 to support + Docker versions earlier than 1.10. +- This document is about the admin guide. To learn how to use GitLab Container + Registry [user documentation](../user/project/container_registry.md). -- [Enable the Container Registry](#enable-the-container-registry) -- [Container Registry domain configuration](#container-registry-domain-configuration) - - [Configure Container Registry under an existing GitLab domain](#configure-container-registry-under-an-existing-gitlab-domain) - - [Configure Container Registry under its own domain](#configure-container-registry-under-its-own-domain) -- [Disable Container Registry site-wide](#disable-container-registry-site-wide) -- [Disable Container Registry per project](#disable-container-registry-per-project) -- [Disable Container Registry for new projects site-wide](#disable-container-registry-for-new-projects-site-wide) -- [Container Registry storage path](#container-registry-storage-path) -- [Container Registry storage driver](#container-registry-storage-driver) -- [Storage limitations](#storage-limitations) -- [Changelog](#changelog) +With the Container Registry integrated into GitLab, every project can have its +own space to store its Docker images. -<!-- END doctoc generated TOC please keep comment here to allow auto update --> +You can read more about the Container Registry at +https://docs.docker.com/registry/introduction/. ## Enable the Container Registry **Omnibus GitLab installations** All you have to do is configure the domain name under which the Container -Registry will listen to. Read [#container-registry-domain-configuration](#container-registry-domain-configuration) +Registry will listen to. Read +[#container-registry-domain-configuration](#container-registry-domain-configuration) and pick one of the two options that fits your case. >**Note:** -The container Registry works under HTTPS by default. Using HTTP is possible +The container registry works under HTTPS by default. Using HTTP is possible but not recommended and out of the scope of this document. Read the [insecure Registry documentation][docker-insecure] if you want to implement this. @@ -47,7 +37,7 @@ implement this. If you have installed GitLab from source: -1. You will have to [install Docker Registry][registry-deploy] by yourself. +1. You will have to [install Registry][registry-deploy] by yourself. 1. After the installation is complete, you will have to configure the Registry's settings in `gitlab.yml` in order to enable it. 1. Use the sample NGINX configuration file that is found under @@ -80,11 +70,13 @@ where: | `issuer` | This should be the same value as configured in Registry's `issuer`. Read the [token auth configuration documentation][token-config]. | >**Note:** -GitLab does not ship with a Registry init file. Hence, [restarting GitLab][restart gitlab] -will not restart the Registry should you modify its settings. Read the upstream -documentation on how to achieve that. +A Registry init file is not shipped with GitLab if you install it from source. +Hence, [restarting GitLab][restart gitlab] will not restart the Registry should +you modify its settings. Read the upstream documentation on how to achieve that. -The Docker Registry configuration will need `container_registry` as the service and `https://gitlab.example.com/jwt/auth` as the realm: +At the absolute minimum, make sure your [Registry configuration][registry-auth] +has `container_registry` as the service and `https://gitlab.example.com/jwt/auth` +as the realm: ``` auth: @@ -275,12 +267,6 @@ Registry application itself. 1. Save the file and [restart GitLab][] for the changes to take effect. -## Disable Container Registry per project - -If Registry is enabled in your GitLab instance, but you don't need it for your -project, you can disable it from your project's settings. Read the user guide -on how to achieve that. - ## Disable Container Registry for new projects site-wide If the Container Registry is enabled, then it will be available on all new @@ -436,6 +422,46 @@ storage: enabled: true ``` +## Change the registry's internal port + +> **Note:** +This is not to be confused with the port that GitLab itself uses to expose +the Registry to the world. + +The Registry server listens on localhost at port `5000` by default, +which is the address for which the Registry server should accept connections. +In the examples below we set the Registry's port to `5001`. + +**Omnibus GitLab** + +1. Open `/etc/gitlab/gitlab.rb` and set `registry['registry_http_addr']`: + + ```ruby + registry['registry_http_addr'] = "localhost:5001" + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +--- + +**Installations from source** + +1. Open the configuration file of your Registry server and edit the + [`http:addr`][registry-http-config] value: + + ``` + http + addr: localhost:5001 + ``` + +1. Save the file and restart the Registry server. + +## Disable Container Registry per project + +If Registry is enabled in your GitLab instance, but you don't need it for your +project, you can disable it from your project's settings. Read the user guide +on how to achieve that. + ## Storage limitations Currently, there is no storage limitation, which means a user can upload an @@ -455,6 +481,8 @@ configurable in future releases. [docker-insecure]: https://docs.docker.com/registry/insecure/ [registry-deploy]: https://docs.docker.com/registry/deploying/ [storage-config]: https://docs.docker.com/registry/configuration/#storage +[registry-http-config]: https://docs.docker.com/registry/configuration/#http +[registry-auth]: https://docs.docker.com/registry/configuration/#auth [token-config]: https://docs.docker.com/registry/configuration/#token [8-8-docs]: https://gitlab.com/gitlab-org/gitlab-ce/blob/8-8-stable/doc/administration/container_registry.md [registry-ssl]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/support/nginx/registry-ssl diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index 7f53915a4d7..b4a953d1ccc 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -13,15 +13,17 @@ override certain values. Variable | Type | Description -------- | ---- | ----------- -`GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation -`GITLAB_HOST` | string | The full URL of the GitLab server (including `http://` or `https://`) -`RAILS_ENV` | string | The Rails environment; can be one of `production`, `development`, `staging` or `test` -`DATABASE_URL` | string | The database URL; is of the form: `postgresql://localhost/blog_development` -`GITLAB_EMAIL_FROM` | string | The e-mail address used in the "From" field in e-mails sent by GitLab -`GITLAB_EMAIL_DISPLAY_NAME` | string | The name used in the "From" field in e-mails sent by GitLab -`GITLAB_EMAIL_REPLY_TO` | string | The e-mail address used in the "Reply-To" field in e-mails sent by GitLab -`GITLAB_UNICORN_MEMORY_MIN` | integer | The minimum memory threshold (in bytes) for the Unicorn worker killer -`GITLAB_UNICORN_MEMORY_MAX` | integer | The maximum memory threshold (in bytes) for the Unicorn worker killer +`GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation +`GITLAB_HOST` | string | The full URL of the GitLab server (including `http://` or `https://`) +`RAILS_ENV` | string | The Rails environment; can be one of `production`, `development`, `staging` or `test` +`DATABASE_URL` | string | The database URL; is of the form: `postgresql://localhost/blog_development` +`GITLAB_EMAIL_FROM` | string | The e-mail address used in the "From" field in e-mails sent by GitLab +`GITLAB_EMAIL_DISPLAY_NAME` | string | The name used in the "From" field in e-mails sent by GitLab +`GITLAB_EMAIL_REPLY_TO` | string | The e-mail address used in the "Reply-To" field in e-mails sent by GitLab +`GITLAB_EMAIL_REPLY_TO` | string | The e-mail address used in the "Reply-To" field in e-mails sent by GitLab +`GITLAB_EMAIL_SUBJECT_SUFFIX` | string | The e-mail subject suffix used in e-mails sent by GitLab +`GITLAB_UNICORN_MEMORY_MIN` | integer | The minimum memory threshold (in bytes) for the Unicorn worker killer +`GITLAB_UNICORN_MEMORY_MAX` | integer | The maximum memory threshold (in bytes) for the Unicorn worker killer ## Complete database variables diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md index 8a881ce8863..137fed35a73 100644 --- a/doc/administration/high_availability/gitlab.md +++ b/doc/administration/high_availability/gitlab.md @@ -101,9 +101,9 @@ need some additional configuration. ```ruby gitlab_shell['secret_token'] = 'fbfb19c355066a9afb030992231c4a363357f77345edd0f2e772359e5be59b02538e1fa6cae8f93f7d23355341cea2b93600dab6d6c3edcdced558fc6d739860' - gitlab_rails['secret_token'] = 'b719fe119132c7810908bba18315259ed12888d4f5ee5430c42a776d840a396799b0a5ef0a801348c8a357f07aa72bbd58e25a84b8f247a25c72f539c7a6c5fa' - gitlab_ci['secret_key_base'] = '6e657410d57c71b4fc3ed0d694e7842b1895a8b401d812c17fe61caf95b48a6d703cb53c112bc01ebd197a85da81b18e29682040e99b4f26594772a4a2c98c6d' - gitlab_ci['db_key_base'] = 'bf2e47b68d6cafaef1d767e628b619365becf27571e10f196f98dc85e7771042b9203199d39aff91fcb6837c8ed83f2a912b278da50999bb11a2fbc0fba52964' + gitlab_rails['otp_key_base'] = 'b719fe119132c7810908bba18315259ed12888d4f5ee5430c42a776d840a396799b0a5ef0a801348c8a357f07aa72bbd58e25a84b8f247a25c72f539c7a6c5fa' + gitlab_rails['secret_key_base'] = '6e657410d57c71b4fc3ed0d694e7842b1895a8b401d812c17fe61caf95b48a6d703cb53c112bc01ebd197a85da81b18e29682040e99b4f26594772a4a2c98c6d' + gitlab_rails['db_key_base'] = 'bf2e47b68d6cafaef1d767e628b619365becf27571e10f196f98dc85e7771042b9203199d39aff91fcb6837c8ed83f2a912b278da50999bb11a2fbc0fba52964' ``` 1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 34b4f1faa94..ad1fa98b63b 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -12,7 +12,7 @@ revisions (to reduce disk space and increase performance) and removing unreachable objects which may have been created from prior invocations of `git add`. -You can find this option under your **[Project] > Settings**. +You can find this option under your **[Project] > Edit Project**. --- diff --git a/doc/administration/img/housekeeping_settings.png b/doc/administration/img/housekeeping_settings.png Binary files differindex f72ad9a45d5..6ebc6205635 100644 --- a/doc/administration/img/housekeeping_settings.png +++ b/doc/administration/img/housekeeping_settings.png diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md new file mode 100644 index 00000000000..771584268d9 --- /dev/null +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -0,0 +1,40 @@ +# GitLab Configuration + +GitLab Performance Monitoring is disabled by default. To enable it and change any of its +settings, navigate to the Admin area in **Settings > Metrics** +(`/admin/application_settings`). + +The minimum required settings you need to set are the InfluxDB host and port. +Make sure _Enable InfluxDB Metrics_ is checked and hit **Save** to save the +changes. + +--- + + + +--- + +Finally, a restart of all GitLab processes is required for the changes to take +effect: + +```bash +# For Omnibus installations +sudo gitlab-ctl restart + +# For installations from source +sudo service gitlab restart +``` + +## Pending Migrations + +When any migrations are pending, the metrics are disabled until the migrations +have been performed. + +--- + +Read more on: + +- [Introduction to GitLab Performance Monitoring](introduction.md) +- [InfluxDB Configuration](influxdb_configuration.md) +- [InfluxDB Schema](influxdb_schema.md) +- [Grafana Install/Configuration](grafana_configuration.md) diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md new file mode 100644 index 00000000000..7947b0fedc4 --- /dev/null +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -0,0 +1,111 @@ +# Grafana Configuration + +[Grafana](http://grafana.org/) is a tool that allows you to visualize time +series metrics through graphs and dashboards. It supports several backend +data stores, including InfluxDB. GitLab writes performance data to InfluxDB +and Grafana will allow you to query InfluxDB to display useful graphs. + +For the easiest installation and configuration, install Grafana on the same +server as InfluxDB. For larger installations, you may want to split out these +services. + +## Installation + +Grafana supplies package repositories (Yum/Apt) for easy installation. +See [Grafana installation documentation](http://docs.grafana.org/installation/) +for detailed steps. + +> **Note**: Before starting Grafana for the first time, set the admin user +and password in `/etc/grafana/grafana.ini`. Otherwise, the default password +will be `admin`. + +## Configuration + +Login as the admin user. Expand the menu by clicking the Grafana logo in the +top left corner. Choose 'Data Sources' from the menu. Then, click 'Add new' +in the top bar. + + + +Fill in the configuration details for the InfluxDB data source. Save and +Test Connection to ensure the configuration is correct. + +- **Name**: InfluxDB +- **Default**: Checked +- **Type**: InfluxDB 0.9.x (Even if you're using InfluxDB 0.10.x) +- **Url**: https://localhost:8086 (Or the remote URL if you've installed InfluxDB +on a separate server) +- **Access**: proxy +- **Database**: gitlab +- **User**: admin (Or the username configured when setting up InfluxDB) +- **Password**: The password configured when you set up InfluxDB + + + +## Apply retention policies and create continuous queries + +If you intend to import the GitLab provided Grafana dashboards, you will need to +set up the right retention policies and continuous queries. The easiest way of +doing this is by using the [influxdb-management](https://gitlab.com/gitlab-org/influxdb-management) +repository. + +To use this repository you must first clone it: + +``` +git clone https://gitlab.com/gitlab-org/influxdb-management.git +cd influxdb-management +``` + +Next you must install the required dependencies: + +``` +gem install bundler +bundle install +``` + +Now you must configure the repository by first copying `.env.example` to `.env` +and then editing the `.env` file to contain the correct InfluxDB settings. Once +configured you can simply run `bundle exec rake` and the InfluxDB database will +be configured for you. + +For more information see the [influxdb-management README](https://gitlab.com/gitlab-org/influxdb-management/blob/master/README.md). + +## Import Dashboards + +You can now import a set of default dashboards that will give you a good +start on displaying useful information. GitLab has published a set of default +[Grafana dashboards][grafana-dashboards] to get you started. Clone the +repository or download a zip/tarball, then follow these steps to import each +JSON file. + +Open the dashboard dropdown menu and click 'Import' + + + +Click 'Choose file' and browse to the location where you downloaded or cloned +the dashboard repository. Pick one of the JSON files to import. + + + +Once the dashboard is imported, be sure to click save icon in the top bar. If +you do not save the dashboard after importing it will be removed when you +navigate away. + + + +Repeat this process for each dashboard you wish to import. + +Alternatively you can automatically import all the dashboards into your Grafana +instance. See the README of the [Grafana dashboards][grafana-dashboards] +repository for more information on this process. + +[grafana-dashboards]: https://gitlab.com/gitlab-org/grafana-dashboards + +--- + +Read more on: + +- [Introduction to GitLab Performance Monitoring](introduction.md) +- [GitLab Configuration](gitlab_configuration.md) +- [InfluxDB Installation/Configuration](influxdb_configuration.md) +- [InfluxDB Schema](influxdb_schema.md) diff --git a/doc/administration/monitoring/performance/img/grafana_dashboard_dropdown.png b/doc/administration/monitoring/performance/img/grafana_dashboard_dropdown.png Binary files differnew file mode 100644 index 00000000000..7e34fad71ce --- /dev/null +++ b/doc/administration/monitoring/performance/img/grafana_dashboard_dropdown.png diff --git a/doc/administration/monitoring/performance/img/grafana_dashboard_import.png b/doc/administration/monitoring/performance/img/grafana_dashboard_import.png Binary files differnew file mode 100644 index 00000000000..f97624365c7 --- /dev/null +++ b/doc/administration/monitoring/performance/img/grafana_dashboard_import.png diff --git a/doc/administration/monitoring/performance/img/grafana_data_source_configuration.png b/doc/administration/monitoring/performance/img/grafana_data_source_configuration.png Binary files differnew file mode 100644 index 00000000000..7d50e4c88c2 --- /dev/null +++ b/doc/administration/monitoring/performance/img/grafana_data_source_configuration.png diff --git a/doc/administration/monitoring/performance/img/grafana_data_source_empty.png b/doc/administration/monitoring/performance/img/grafana_data_source_empty.png Binary files differnew file mode 100644 index 00000000000..aa39a53acae --- /dev/null +++ b/doc/administration/monitoring/performance/img/grafana_data_source_empty.png diff --git a/doc/administration/monitoring/performance/img/grafana_save_icon.png b/doc/administration/monitoring/performance/img/grafana_save_icon.png Binary files differnew file mode 100644 index 00000000000..c740e33cd1c --- /dev/null +++ b/doc/administration/monitoring/performance/img/grafana_save_icon.png diff --git a/doc/administration/monitoring/performance/img/metrics_gitlab_configuration_settings.png b/doc/administration/monitoring/performance/img/metrics_gitlab_configuration_settings.png Binary files differnew file mode 100644 index 00000000000..db396423e30 --- /dev/null +++ b/doc/administration/monitoring/performance/img/metrics_gitlab_configuration_settings.png diff --git a/doc/administration/monitoring/performance/img/request_profile_result.png b/doc/administration/monitoring/performance/img/request_profile_result.png Binary files differnew file mode 100644 index 00000000000..73e2fdcab67 --- /dev/null +++ b/doc/administration/monitoring/performance/img/request_profile_result.png diff --git a/doc/administration/monitoring/performance/img/request_profiling_token.png b/doc/administration/monitoring/performance/img/request_profiling_token.png Binary files differnew file mode 100644 index 00000000000..04d87567816 --- /dev/null +++ b/doc/administration/monitoring/performance/img/request_profiling_token.png diff --git a/doc/administration/monitoring/performance/influxdb_configuration.md b/doc/administration/monitoring/performance/influxdb_configuration.md new file mode 100644 index 00000000000..c30cd2950d8 --- /dev/null +++ b/doc/administration/monitoring/performance/influxdb_configuration.md @@ -0,0 +1,193 @@ +# InfluxDB Configuration + +The default settings provided by [InfluxDB] are not sufficient for a high traffic +GitLab environment. The settings discussed in this document are based on the +settings GitLab uses for GitLab.com, depending on your own needs you may need to +further adjust them. + +If you are intending to run InfluxDB on the same server as GitLab, make sure +you have plenty of RAM since InfluxDB can use quite a bit depending on traffic. + +Unless you are going with a budget setup, it's advised to run it separately. + +## Requirements + +- InfluxDB 0.9.5 or newer +- A fairly modern version of Linux +- At least 4GB of RAM +- At least 10GB of storage for InfluxDB data + +Note that the RAM and storage requirements can differ greatly depending on the +amount of data received/stored. To limit the amount of stored data users can +look into [InfluxDB Retention Policies][influxdb-retention]. + +## Installation + +Installing InfluxDB is out of the scope of this document. Please refer to the +[InfluxDB documentation]. + +## InfluxDB Server Settings + +Since InfluxDB has many settings that users may wish to customize themselves +(e.g. what port to run InfluxDB on), we'll only cover the essentials. + +The configuration file in question is usually located at +`/etc/influxdb/influxdb.conf`. Whenever you make a change in this file, +InfluxDB needs to be restarted. + +### Storage Engine + +InfluxDB comes with different storage engines and as of InfluxDB 0.9.5 a new +storage engine is available, called [TSM Tree]. All users **must** use the new +`tsm1` storage engine as this [will be the default engine][tsm1-commit] in +upcoming InfluxDB releases. + +Make sure you have the following in your configuration file: + +``` +[data] + dir = "/var/lib/influxdb/data" + engine = "tsm1" +``` + +### Admin Panel + +Production environments should have the InfluxDB admin panel **disabled**. This +feature can be disabled by adding the following to your InfluxDB configuration +file: + +``` +[admin] + enabled = false +``` + +### HTTP + +HTTP is required when using the [InfluxDB CLI] or other tools such as Grafana, +thus it should be enabled. When enabling make sure to _also_ enable +authentication: + +``` +[http] + enabled = true + auth-enabled = true +``` + +_**Note:** Before you enable authentication, you might want to [create an +admin user](#create-a-new-admin-user)._ + +### UDP + +GitLab writes data to InfluxDB via UDP and thus this must be enabled. Enabling +UDP can be done using the following settings: + +``` +[[udp]] + enabled = true + bind-address = ":8089" + database = "gitlab" + batch-size = 1000 + batch-pending = 5 + batch-timeout = "1s" + read-buffer = 209715200 +``` + +This does the following: + +1. Enable UDP and bind it to port 8089 for all addresses. +2. Store any data received in the "gitlab" database. +3. Define a batch of points to be 1000 points in size and allow a maximum of + 5 batches _or_ flush them automatically after 1 second. +4. Define a UDP read buffer size of 200 MB. + +One of the most important settings here is the UDP read buffer size as if this +value is set too low, packets will be dropped. You must also make sure the OS +buffer size is set to the same value, the default value is almost never enough. + +To set the OS buffer size to 200 MB, on Linux you can run the following command: + +```bash +sysctl -w net.core.rmem_max=209715200 +``` + +To make this permanent, add the following to `/etc/sysctl.conf` and restart the +server: + +```bash +net.core.rmem_max=209715200 +``` + +It is **very important** to make sure the buffer sizes are large enough to +handle all data sent to InfluxDB as otherwise you _will_ lose data. The above +buffer sizes are based on the traffic for GitLab.com. Depending on the amount of +traffic, users may be able to use a smaller buffer size, but we highly recommend +using _at least_ 100 MB. + +When enabling UDP, users should take care to not expose the port to the public, +as doing so will allow anybody to write data into your InfluxDB database (as +[InfluxDB's UDP protocol][udp] doesn't support authentication). We recommend either +whitelisting the allowed IP addresses/ranges, or setting up a VLAN and only +allowing traffic from members of said VLAN. + +## Create a new admin user + +If you want to [enable authentication](#http), you might want to [create an +admin user][influx-admin]: + +``` +influx -execute "CREATE USER jeff WITH PASSWORD '1234' WITH ALL PRIVILEGES" +``` + +## Create the `gitlab` database + +Once you get InfluxDB up and running, you need to create a database for GitLab. +Make sure you have changed the [storage engine](#storage-engine) to `tsm1` +before creating a database. + +_**Note:** If you [created an admin user](#create-a-new-admin-user) and enabled +[HTTP authentication](#http), remember to append the username (`-username <username>`) +and password (`-password <password>`) you set earlier to the commands below._ + +Run the following command to create a database named `gitlab`: + +```bash +influx -execute 'CREATE DATABASE gitlab' +``` + +The name **must** be `gitlab`, do not use any other name. + +Next, make sure that the database was successfully created: + +```bash +influx -execute 'SHOW DATABASES' +``` + +The output should be similar to: + +``` +name: databases +--------------- +name +_internal +gitlab +``` + +That's it! Now your GitLab instance should send data to InfluxDB. + +--- + +Read more on: + +- [Introduction to GitLab Performance Monitoring](introduction.md) +- [GitLab Configuration](gitlab_configuration.md) +- [InfluxDB Schema](influxdb_schema.md) +- [Grafana Install/Configuration](grafana_configuration.md) + +[influxdb-retention]: https://docs.influxdata.com/influxdb/v0.9/query_language/database_management/#retention-policy-management +[influxdb documentation]: https://docs.influxdata.com/influxdb/v0.9/ +[influxdb cli]: https://docs.influxdata.com/influxdb/v0.9/tools/shell/ +[udp]: https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/ +[influxdb]: https://influxdata.com/time-series-platform/influxdb/ +[tsm tree]: https://influxdata.com/blog/new-storage-engine-time-structured-merge-tree/ +[tsm1-commit]: https://github.com/influxdata/influxdb/commit/15d723dc77651bac83e09e2b1c94be480966cb0d +[influx-admin]: https://docs.influxdata.com/influxdb/v0.9/administration/authentication_and_authorization/#create-a-new-admin-user diff --git a/doc/administration/monitoring/performance/influxdb_schema.md b/doc/administration/monitoring/performance/influxdb_schema.md new file mode 100644 index 00000000000..eff0e29f58d --- /dev/null +++ b/doc/administration/monitoring/performance/influxdb_schema.md @@ -0,0 +1,97 @@ +# InfluxDB Schema + +The following measurements are currently stored in InfluxDB: + +- `PROCESS_file_descriptors` +- `PROCESS_gc_statistics` +- `PROCESS_memory_usage` +- `PROCESS_method_calls` +- `PROCESS_object_counts` +- `PROCESS_transactions` +- `PROCESS_views` +- `events` + +Here, `PROCESS` is replaced with either `rails` or `sidekiq` depending on the +process type. In all series, any form of duration is stored in milliseconds. + +## PROCESS_file_descriptors + +This measurement contains the number of open file descriptors over time. The +value field `value` contains the number of descriptors. + +## PROCESS_gc_statistics + +This measurement contains Ruby garbage collection statistics such as the amount +of minor/major GC runs (relative to the last sampling interval), the time spent +in garbage collection cycles, and all fields/values returned by `GC.stat`. + +## PROCESS_memory_usage + +This measurement contains the process' memory usage (in bytes) over time. The +value field `value` contains the number of bytes. + +## PROCESS_method_calls + +This measurement contains the methods called during a transaction along with +their duration, and a name of the transaction action that invoked the method (if +available). The method call duration is stored in the value field `duration`, +while the method name is stored in the tag `method`. The tag `action` contains +the full name of the transaction action. Both the `method` and `action` fields +are in the following format: + +``` +ClassName#method_name +``` + +For example, a method called by the `show` method in the `UsersController` class +would have `action` set to `UsersController#show`. + +## PROCESS_object_counts + +This measurement is used to store retained Ruby objects (per class) and the +amount of retained objects. The number of objects is stored in the `count` value +field while the class name is stored in the `type` tag. + +## PROCESS_transactions + +This measurement is used to store basic transaction details such as the time it +took to complete a transaction, how much time was spent in SQL queries, etc. The +following value fields are available: + +| Value | Description | +| ----- | ----------- | +| `duration` | The total duration of the transaction | +| `allocated_memory` | The amount of bytes allocated while the transaction was running. This value is only reliable when using single-threaded application servers | +| `method_duration` | The total time spent in method calls | +| `sql_duration` | The total time spent in SQL queries | +| `view_duration` | The total time spent in views | + +## PROCESS_views + +This measurement is used to store view rendering timings for a transaction. The +following value fields are available: + +| Value | Description | +| ----- | ----------- | +| `duration` | The rendering time of the view | +| `view` | The path of the view, relative to the application's root directory | + +The `action` tag contains the action name of the transaction that rendered the +view. + +## events + +This measurement is used to store generic events such as the number of Git +pushes, Emails sent, etc. Each point in this measurement has a single value +field called `count`. The value of this field is simply set to `1`. Each point +also has at least one tag: `event`. This tag's value is set to the event name. +Depending on the event type additional tags may be available as well. + +--- + +Read more on: + +- [Introduction to GitLab Performance Monitoring](introduction.md) +- [GitLab Configuration](gitlab_configuration.md) +- [InfluxDB Configuration](influxdb_configuration.md) +- [Grafana Install/Configuration](grafana_configuration.md) diff --git a/doc/administration/monitoring/performance/introduction.md b/doc/administration/monitoring/performance/introduction.md new file mode 100644 index 00000000000..79904916b7e --- /dev/null +++ b/doc/administration/monitoring/performance/introduction.md @@ -0,0 +1,65 @@ +# GitLab Performance Monitoring + +GitLab comes with its own application performance measuring system as of GitLab +8.4, simply called "GitLab Performance Monitoring". GitLab Performance Monitoring is available in both the +Community and Enterprise editions. + +Apart from this introduction, you are advised to read through the following +documents in order to understand and properly configure GitLab Performance Monitoring: + +- [GitLab Configuration](gitlab_configuration.md) +- [InfluxDB Install/Configuration](influxdb_configuration.md) +- [InfluxDB Schema](influxdb_schema.md) +- [Grafana Install/Configuration](grafana_configuration.md) + +## Introduction to GitLab Performance Monitoring + +GitLab Performance Monitoring makes it possible to measure a wide variety of statistics +including (but not limited to): + +- The time it took to complete a transaction (a web request or Sidekiq job). +- The time spent in running SQL queries and rendering HAML views. +- The time spent executing (instrumented) Ruby methods. +- Ruby object allocations, and retained objects in particular. +- System statistics such as the process' memory usage and open file descriptors. +- Ruby garbage collection statistics. + +Metrics data is written to [InfluxDB][influxdb] over [UDP][influxdb-udp]. Stored +data can be visualized using [Grafana][grafana] or any other application that +supports reading data from InfluxDB. Alternatively data can be queried using the +InfluxDB CLI. + +## Metric Types + +Two types of metrics are collected: + +1. Transaction specific metrics. +1. Sampled metrics, collected at a certain interval in a separate thread. + +### Transaction Metrics + +Transaction metrics are metrics that can be associated with a single +transaction. This includes statistics such as the transaction duration, timings +of any executed SQL queries, time spent rendering HAML views, etc. These metrics +are collected for every Rack request and Sidekiq job processed. + +### Sampled Metrics + +Sampled metrics are metrics that can't be associated with a single transaction. +Examples include garbage collection statistics and retained Ruby objects. These +metrics are collected at a regular interval. This interval is made up out of two +parts: + +1. A user defined interval. +1. A randomly generated offset added on top of the interval, the same offset + can't be used twice in a row. + +The actual interval can be anywhere between a half of the defined interval and a +half above the interval. For example, for a user defined interval of 15 seconds +the actual interval can be anywhere between 7.5 and 22.5. The interval is +re-generated for every sampling run instead of being generated once and re-used +for the duration of the process' lifetime. + +[influxdb]: https://influxdata.com/time-series-platform/influxdb/ +[influxdb-udp]: https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/ +[grafana]: http://grafana.org/ diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md new file mode 100644 index 00000000000..c358dfbead2 --- /dev/null +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -0,0 +1,16 @@ +# Request Profiling + +## Procedure +1. Grab the profiling token from `Monitoring > Requests Profiles` admin page +(highlighted in a blue in the image below). + +1. Pass the header `X-Profile-Token: <token>` to the request you want to profile. You can use any of these tools + * [ModHeader](https://chrome.google.com/webstore/detail/modheader/idgpnmonknjnojddfkpgkljpfnnfcklj) Chrome extension + * [Modify Headers](https://addons.mozilla.org/en-US/firefox/addon/modify-headers/) Firefox extension + * `curl --header 'X-Profile-Token: <token>' https://gitlab.example.com/group/project` +1. Once request is finished (which will take a little longer than usual), you can +view the profiling output from `Monitoring > Requests Profiles` admin page. + + +## Cleaning up +Profiling output will be cleared out every day via a Sidekiq worker. diff --git a/doc/administration/operations.md b/doc/administration/operations.md new file mode 100644 index 00000000000..4b582d16b64 --- /dev/null +++ b/doc/administration/operations.md @@ -0,0 +1,6 @@ +# GitLab operations + +- [Sidekiq MemoryKiller](operations/sidekiq_memory_killer.md) +- [Cleaning up Redis sessions](operations/cleaning_up_redis_sessions.md) +- [Understanding Unicorn and unicorn-worker-killer](operations/unicorn.md) +- [Moving repositories to a new location](operations/moving_repositories.md) diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md new file mode 100644 index 00000000000..93521e976d5 --- /dev/null +++ b/doc/administration/operations/cleaning_up_redis_sessions.md @@ -0,0 +1,52 @@ +# Cleaning up stale Redis sessions + +Since version 6.2, GitLab stores web user sessions as key-value pairs in Redis. +Prior to GitLab 7.3, user sessions did not automatically expire from Redis. If +you have been running a large GitLab server (thousands of users) since before +GitLab 7.3 we recommend cleaning up stale sessions to compact the Redis +database after you upgrade to GitLab 7.3. You can also perform a cleanup while +still running GitLab 7.2 or older, but in that case new stale sessions will +start building up again after you clean up. + +In GitLab versions prior to 7.3.0, the session keys in Redis are 16-byte +hexadecimal values such as '976aa289e2189b17d7ef525a6702ace9'. Starting with +GitLab 7.3.0, the keys are +prefixed with 'session:gitlab:', so they would look like +'session:gitlab:976aa289e2189b17d7ef525a6702ace9'. Below we describe how to +remove the keys in the old format. + +First we define a shell function with the proper Redis connection details. + +``` +rcli() { + # This example works for Omnibus installations of GitLab 7.3 or newer. For an + # installation from source you will have to change the socket path and the + # path to redis-cli. + sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket "$@" +} + +# test the new shell function; the response should be PONG +rcli ping +``` + +Now we do a search to see if there are any session keys in the old format for +us to clean up. + +``` +# returns the number of old-format session keys in Redis +rcli keys '*' | grep '^[a-f0-9]\{32\}$' | wc -l +``` + +If the number is larger than zero, you can proceed to expire the keys from +Redis. If the number is zero there is nothing to clean up. + +``` +# Tell Redis to expire each matched key after 600 seconds. +rcli keys '*' | grep '^[a-f0-9]\{32\}$' | awk '{ print "expire", $0, 600 }' | rcli +# This will print '(integer) 1' for each key that gets expired. +``` + +Over the next 15 minutes (10 minutes expiry time plus 5 minutes Redis +background save interval) your Redis database will be compacted. If you are +still using GitLab 7.2, users who are not clicking around in GitLab during the +10 minute expiry window will be signed out of GitLab. diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md new file mode 100644 index 00000000000..54adb99386a --- /dev/null +++ b/doc/administration/operations/moving_repositories.md @@ -0,0 +1,180 @@ +# Moving repositories managed by GitLab + +Sometimes you need to move all repositories managed by GitLab to +another filesystem or another server. In this document we will look +at some of the ways you can copy all your repositories from +`/var/opt/gitlab/git-data/repositories` to `/mnt/gitlab/repositories`. + +We will look at three scenarios: the target directory is empty, the +target directory contains an outdated copy of the repositories, and +how to deal with thousands of repositories. + +**Each of the approaches we list can/will overwrite data in the +target directory `/mnt/gitlab/repositories`. Do not mix up the +source and the target.** + +## Target directory is empty: use a tar pipe + +If the target directory `/mnt/gitlab/repositories` is empty the +simplest thing to do is to use a tar pipe. This method has low +overhead and tar is almost always already installed on your system. +However, it is not possible to resume an interrupted tar pipe: if +that happens then all data must be copied again. + +``` +# As the git user +tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\ + tar -C /mnt/gitlab/repositories -xf - +``` + +If you want to see progress, replace `-xf` with `-xvf`. + +### Tar pipe to another server + +You can also use a tar pipe to copy data to another server. If your +'git' user has SSH access to the newserver as 'git@newserver', you +can pipe the data through SSH. + +``` +# As the git user +tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\ + ssh git@newserver tar -C /mnt/gitlab/repositories -xf - +``` + +If you want to compress the data before it goes over the network +(which will cost you CPU cycles) you can replace `ssh` with `ssh -C`. + +## The target directory contains an outdated copy of the repositories: use rsync + +If the target directory already contains a partial / outdated copy +of the repositories it may be wasteful to copy all the data again +with tar. In this scenario it is better to use rsync. This utility +is either already installed on your system or easily installable +via apt, yum etc. + +``` +# As the 'git' user +rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ + /mnt/gitlab/repositories +``` + +The `/.` in the command above is very important, without it you can +easily get the wrong directory structure in the target directory. +If you want to see progress, replace `-a` with `-av`. + +### Single rsync to another server + +If the 'git' user on your source system has SSH access to the target +server you can send the repositories over the network with rsync. + +``` +# As the 'git' user +rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ + git@newserver:/mnt/gitlab/repositories +``` + +## Thousands of Git repositories: use one rsync per repository + +Every time you start an rsync job it has to inspect all files in +the source directory, all files in the target directory, and then +decide what files to copy or not. If the source or target directory +has many contents this startup phase of rsync can become a burden +for your GitLab server. In cases like this you can make rsync's +life easier by dividing its work in smaller pieces, and sync one +repository at a time. + +In addition to rsync we will use [GNU +Parallel](http://www.gnu.org/software/parallel/). This utility is +not included in GitLab so you need to install it yourself with apt +or yum. Also note that the GitLab scripts we used below were added +in GitLab 8.1. + +** This process does not clean up repositories at the target location that no +longer exist at the source. ** If you start using your GitLab instance with +`/mnt/gitlab/repositories`, you need to run `gitlab-rake gitlab:cleanup:repos` +after switching to the new repository storage directory. + +### Parallel rsync for all repositories known to GitLab + +This will sync repositories with 10 rsync processes at a time. We keep +track of progress so that the transfer can be restarted if necessary. + +First we create a new directory, owned by 'git', to hold transfer +logs. We assume the directory is empty before we start the transfer +procedure, and that we are the only ones writing files in it. + +``` +# Omnibus +sudo mkdir /var/opt/gitlab/transfer-logs +sudo chown git:git /var/opt/gitlab/transfer-logs + +# Source +sudo -u git -H mkdir /home/git/transfer-logs +``` + +We seed the process with a list of the directories we want to copy. + +``` +# Omnibus +sudo -u git sh -c 'gitlab-rake gitlab:list_repos > /var/opt/gitlab/transfer-logs/all-repos-$(date +%s).txt' + +# Source +cd /home/git/gitlab +sudo -u git -H sh -c 'bundle exec rake gitlab:list_repos > /home/git/transfer-logs/all-repos-$(date +%s).txt' +``` + +Now we can start the transfer. The command below is idempotent, and +the number of jobs done by GNU Parallel should converge to zero. If it +does not some repositories listed in all-repos-1234.txt may have been +deleted/renamed before they could be copied. + +``` +# Omnibus +sudo -u git sh -c ' +cat /var/opt/gitlab/transfer-logs/* | sort | uniq -u |\ + /usr/bin/env JOBS=10 \ + /opt/gitlab/embedded/service/gitlab-rails/bin/parallel-rsync-repos \ + /var/opt/gitlab/transfer-logs/success-$(date +%s).log \ + /var/opt/gitlab/git-data/repositories \ + /mnt/gitlab/repositories +' + +# Source +cd /home/git/gitlab +sudo -u git -H sh -c ' +cat /home/git/transfer-logs/* | sort | uniq -u |\ + /usr/bin/env JOBS=10 \ + bin/parallel-rsync-repos \ + /home/git/transfer-logs/success-$(date +%s).log \ + /home/git/repositories \ + /mnt/gitlab/repositories +` +``` + +### Parallel rsync only for repositories with recent activity + +Suppose you have already done one sync that started after 2015-10-1 12:00 UTC. +Then you might only want to sync repositories that were changed via GitLab +_after_ that time. You can use the 'SINCE' variable to tell 'rake +gitlab:list_repos' to only print repositories with recent activity. + +``` +# Omnibus +sudo gitlab-rake gitlab:list_repos SINCE='2015-10-1 12:00 UTC' |\ + sudo -u git \ + /usr/bin/env JOBS=10 \ + /opt/gitlab/embedded/service/gitlab-rails/bin/parallel-rsync-repos \ + success-$(date +%s).log \ + /var/opt/gitlab/git-data/repositories \ + /mnt/gitlab/repositories + +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:list_repos SINCE='2015-10-1 12:00 UTC' |\ + sudo -u git -H \ + /usr/bin/env JOBS=10 \ + bin/parallel-rsync-repos \ + success-$(date +%s).log \ + /home/git/repositories \ + /mnt/gitlab/repositories +``` diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md new file mode 100644 index 00000000000..b5e78348989 --- /dev/null +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -0,0 +1,40 @@ +# Sidekiq MemoryKiller + +The GitLab Rails application code suffers from memory leaks. For web requests +this problem is made manageable using +[unicorn-worker-killer](https://github.com/kzk/unicorn-worker-killer) which +restarts Unicorn worker processes in between requests when needed. The Sidekiq +MemoryKiller applies the same approach to the Sidekiq processes used by GitLab +to process background jobs. + +Unlike unicorn-worker-killer, which is enabled by default for all GitLab +installations since GitLab 6.4, the Sidekiq MemoryKiller is enabled by default +_only_ for Omnibus packages. The reason for this is that the MemoryKiller +relies on Runit to restart Sidekiq after a memory-induced shutdown and GitLab +installations from source do not all use Runit or an equivalent. + +With the default settings, the MemoryKiller will cause a Sidekiq restart no +more often than once every 15 minutes, with the restart causing about one +minute of delay for incoming background jobs. + +## Configuring the MemoryKiller + +The MemoryKiller is controlled using environment variables. + +- `SIDEKIQ_MEMORY_KILLER_MAX_RSS`: if this variable is set, and its value is + greater than 0, then after each Sidekiq job, the MemoryKiller will check the + RSS of the Sidekiq process that executed the job. If the RSS of the Sidekiq + process (expressed in kilobytes) exceeds SIDEKIQ_MEMORY_KILLER_MAX_RSS, a + delayed shutdown is triggered. The default value for Omnibus packages is set + [in the omnibus-gitlab + repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb). +- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`: defaults 900 seconds (15 minutes). When + a shutdown is triggered, the Sidekiq process will keep working normally for + another 15 minutes. +- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. When the grace + time has expired, the MemoryKiller tells Sidekiq to stop accepting new jobs. + Existing jobs get 30 seconds to finish. After that, the MemoryKiller tells + Sidekiq to shut down, and an external supervision mechanism (e.g. Runit) must + restart Sidekiq. +- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_SIGNAL`: defaults to `SIGKILL`. The name of + the final signal sent to the Sidekiq process when we want it to shut down. diff --git a/doc/administration/operations/unicorn.md b/doc/administration/operations/unicorn.md new file mode 100644 index 00000000000..bad61151bda --- /dev/null +++ b/doc/administration/operations/unicorn.md @@ -0,0 +1,86 @@ +# Understanding Unicorn and unicorn-worker-killer + +## Unicorn + +GitLab uses [Unicorn](http://unicorn.bogomips.org/), a pre-forking Ruby web +server, to handle web requests (web browsers and Git HTTP clients). Unicorn is +a daemon written in Ruby and C that can load and run a Ruby on Rails +application; in our case the Rails application is GitLab Community Edition or +GitLab Enterprise Edition. + +Unicorn has a multi-process architecture to make better use of available CPU +cores (processes can run on different cores) and to have stronger fault +tolerance (most failures stay isolated in only one process and cannot take down +GitLab entirely). On startup, the Unicorn 'master' process loads a clean Ruby +environment with the GitLab application code, and then spawns 'workers' which +inherit this clean initial environment. The 'master' never handles any +requests, that is left to the workers. The operating system network stack +queues incoming requests and distributes them among the workers. + +In a perfect world, the master would spawn its pool of workers once, and then +the workers handle incoming web requests one after another until the end of +time. In reality, worker processes can crash or time out: if the master notices +that a worker takes too long to handle a request it will terminate the worker +process with SIGKILL ('kill -9'). No matter how the worker process ended, the +master process will replace it with a new 'clean' process again. Unicorn is +designed to be able to replace 'crashed' workers without dropping user +requests. + +This is what a Unicorn worker timeout looks like in `unicorn_stderr.log`. The +master process has PID 56227 below. + +``` +[2015-06-05T10:58:08.660325 #56227] ERROR -- : worker=10 PID:53009 timeout (61s > 60s), killing +[2015-06-05T10:58:08.699360 #56227] ERROR -- : reaped #<Process::Status: pid 53009 SIGKILL (signal 9)> worker=10 +[2015-06-05T10:58:08.708141 #62538] INFO -- : worker=10 spawned pid=62538 +[2015-06-05T10:58:08.708824 #62538] INFO -- : worker=10 ready +``` + +### Tunables + +The main tunables for Unicorn are the number of worker processes and the +request timeout after which the Unicorn master terminates a worker process. +See the [omnibus-gitlab Unicorn settings +documentation](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/unicorn.md) +if you want to adjust these settings. + +## unicorn-worker-killer + +GitLab has memory leaks. These memory leaks manifest themselves in long-running +processes, such as Unicorn workers. (The Unicorn master process is not known to +leak memory, probably because it does not handle user requests.) + +To make these memory leaks manageable, GitLab comes with the +[unicorn-worker-killer gem](https://github.com/kzk/unicorn-worker-killer). This +gem [monkey-patches](https://en.wikipedia.org/wiki/Monkey_patch) the Unicorn +workers to do a memory self-check after every 16 requests. If the memory of the +Unicorn worker exceeds a pre-set limit then the worker process exits. The +Unicorn master then automatically replaces the worker process. + +This is a robust way to handle memory leaks: Unicorn is designed to handle +workers that 'crash' so no user requests will be dropped. The +unicorn-worker-killer gem is designed to only terminate a worker process _in +between requests_, so no user requests are affected. + +This is what a Unicorn worker memory restart looks like in unicorn_stderr.log. +You see that worker 4 (PID 125918) is inspecting itself and decides to exit. +The threshold memory value was 254802235 bytes, about 250MB. With GitLab this +threshold is a random value between 200 and 250 MB. The master process (PID +117565) then reaps the worker process and spawns a new 'worker 4' with PID +127549. + +``` +[2015-06-05T12:07:41.828374 #125918] WARN -- : #<Unicorn::HttpServer:0x00000002734770>: worker (pid: 125918) exceeds memory limit (256413696 bytes > 254802235 bytes) +[2015-06-05T12:07:41.828472 #125918] WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 125918) alive: 23 sec (trial 1) +[2015-06-05T12:07:42.025916 #117565] INFO -- : reaped #<Process::Status: pid 125918 exit 0> worker=4 +[2015-06-05T12:07:42.034527 #127549] INFO -- : worker=4 spawned pid=127549 +[2015-06-05T12:07:42.035217 #127549] INFO -- : worker=4 ready +``` + +One other thing that stands out in the log snippet above, taken from +GitLab.com, is that 'worker 4' was serving requests for only 23 seconds. This +is a normal value for our current GitLab.com setup and traffic. + +The high frequency of Unicorn memory restarts on some GitLab sites can be a +source of confusion for administrators. Usually they are a [red +herring](https://en.wikipedia.org/wiki/Red_herring). diff --git a/doc/administration/reply_by_email.md b/doc/administration/reply_by_email.md new file mode 100644 index 00000000000..5a9a1582877 --- /dev/null +++ b/doc/administration/reply_by_email.md @@ -0,0 +1,302 @@ +# Reply by email + +GitLab can be set up to allow users to comment on issues and merge requests by +replying to notification emails. + +## Requirement + +Reply by email requires an IMAP-enabled email account. GitLab allows you to use +three strategies for this feature: +- using email sub-addressing +- using a dedicated email address +- using a catch-all mailbox + +### Email sub-addressing + +**If your provider or server supports email sub-addressing, we recommend using it.** + +[Sub-addressing](https://en.wikipedia.org/wiki/Email_address#Sub-addressing) is +a feature where any email to `user+some_arbitrary_tag@example.com` will end up +in the mailbox for `user@example.com`, and is supported by providers such as +Gmail, Google Apps, Yahoo! Mail, Outlook.com and iCloud, as well as the Postfix +mail server which you can run on-premises. + +### Dedicated email address + +This solution is really simple to set up: you just have to create an email +address dedicated to receive your users' replies to GitLab notifications. + +### Catch-all mailbox + +A [catch-all mailbox](https://en.wikipedia.org/wiki/Catch-all) for a domain will +"catch all" the emails addressed to the domain that do not exist in the mail +server. + +## How it works? + +### 1. GitLab sends a notification email + +When GitLab sends a notification and Reply by email is enabled, the `Reply-To` +header is set to the address defined in your GitLab configuration, with the +`%{key}` placeholder (if present) replaced by a specific "reply key". In +addition, this "reply key" is also added to the `References` header. + +### 2. You reply to the notification email + +When you reply to the notification email, your email client will: + +- send the email to the `Reply-To` address it got from the notification email +- set the `In-Reply-To` header to the value of the `Message-ID` header from the + notification email +- set the `References` header to the value of the `Message-ID` plus the value of + the notification email's `References` header. + +### 3. GitLab receives your reply to the notification email + +When GitLab receives your reply, it will look for the "reply key" in the +following headers, in this order: + +1. the `To` header +1. the `References` header + +If it finds a reply key, it will be able to leave your reply as a comment on +the entity the notification was about (issue, merge request, commit...). + +For more details about the `Message-ID`, `In-Reply-To`, and `References headers`, +please consult [RFC 5322](https://tools.ietf.org/html/rfc5322#section-3.6.4). + +## Set it up + +If you want to use Gmail / Google Apps with Reply by email, make sure you have +[IMAP access enabled](https://support.google.com/mail/troubleshooter/1668960?hl=en#ts=1665018) +and [allowed less secure apps to access the account](https://support.google.com/accounts/answer/6010255). + +To set up a basic Postfix mail server with IMAP access on Ubuntu, follow +[these instructions](./postfix.md). + +### Omnibus package installations + +1. Find the `incoming_email` section in `/etc/gitlab/gitlab.rb`, enable the + feature and fill in the details for your specific IMAP server and email account: + + ```ruby + # Configuration for Postfix mail server, assumes mailbox incoming@gitlab.example.com + gitlab_rails['incoming_email_enabled'] = true + + # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. + # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + gitlab_rails['incoming_email_address'] = "incoming+%{key}@gitlab.example.com" + + # Email account username + # With third party providers, this is usually the full email address. + # With self-hosted email servers, this is usually the user part of the email address. + gitlab_rails['incoming_email_email'] = "incoming" + # Email account password + gitlab_rails['incoming_email_password'] = "[REDACTED]" + + # IMAP server host + gitlab_rails['incoming_email_host'] = "gitlab.example.com" + # IMAP server port + gitlab_rails['incoming_email_port'] = 143 + # Whether the IMAP server uses SSL + gitlab_rails['incoming_email_ssl'] = false + # Whether the IMAP server uses StartTLS + gitlab_rails['incoming_email_start_tls'] = false + + # The mailbox where incoming mail will end up. Usually "inbox". + gitlab_rails['incoming_email_mailbox_name'] = "inbox" + ``` + + ```ruby + # Configuration for Gmail / Google Apps, assumes mailbox gitlab-incoming@gmail.com + gitlab_rails['incoming_email_enabled'] = true + + # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. + # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + gitlab_rails['incoming_email_address'] = "gitlab-incoming+%{key}@gmail.com" + + # Email account username + # With third party providers, this is usually the full email address. + # With self-hosted email servers, this is usually the user part of the email address. + gitlab_rails['incoming_email_email'] = "gitlab-incoming@gmail.com" + # Email account password + gitlab_rails['incoming_email_password'] = "[REDACTED]" + + # IMAP server host + gitlab_rails['incoming_email_host'] = "imap.gmail.com" + # IMAP server port + gitlab_rails['incoming_email_port'] = 993 + # Whether the IMAP server uses SSL + gitlab_rails['incoming_email_ssl'] = true + # Whether the IMAP server uses StartTLS + gitlab_rails['incoming_email_start_tls'] = false + + # The mailbox where incoming mail will end up. Usually "inbox". + gitlab_rails['incoming_email_mailbox_name'] = "inbox" + ``` + +1. Reconfigure GitLab and restart mailroom for the changes to take effect: + + ```sh + sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart mailroom + ``` + +1. Verify that everything is configured correctly: + + ```sh + sudo gitlab-rake gitlab:incoming_email:check + ``` + +1. Reply by email should now be working. + +### Installations from source + +1. Go to the GitLab installation directory: + + ```sh + cd /home/git/gitlab + ``` + +1. Find the `incoming_email` section in `config/gitlab.yml`, enable the feature + and fill in the details for your specific IMAP server and email account: + + ```sh + sudo editor config/gitlab.yml + ``` + + ```yaml + # Configuration for Postfix mail server, assumes mailbox incoming@gitlab.example.com + incoming_email: + enabled: true + + # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. + # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + address: "incoming+%{key}@gitlab.example.com" + + # Email account username + # With third party providers, this is usually the full email address. + # With self-hosted email servers, this is usually the user part of the email address. + user: "incoming" + # Email account password + password: "[REDACTED]" + + # IMAP server host + host: "gitlab.example.com" + # IMAP server port + port: 143 + # Whether the IMAP server uses SSL + ssl: false + # Whether the IMAP server uses StartTLS + start_tls: false + + # The mailbox where incoming mail will end up. Usually "inbox". + mailbox: "inbox" + ``` + + ```yaml + # Configuration for Gmail / Google Apps, assumes mailbox gitlab-incoming@gmail.com + incoming_email: + enabled: true + + # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. + # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + address: "gitlab-incoming+%{key}@gmail.com" + + # Email account username + # With third party providers, this is usually the full email address. + # With self-hosted email servers, this is usually the user part of the email address. + user: "gitlab-incoming@gmail.com" + # Email account password + password: "[REDACTED]" + + # IMAP server host + host: "imap.gmail.com" + # IMAP server port + port: 993 + # Whether the IMAP server uses SSL + ssl: true + # Whether the IMAP server uses StartTLS + start_tls: false + + # The mailbox where incoming mail will end up. Usually "inbox". + mailbox: "inbox" + ``` + +1. Enable `mail_room` in the init script at `/etc/default/gitlab`: + + ```sh + sudo mkdir -p /etc/default + echo 'mail_room_enabled=true' | sudo tee -a /etc/default/gitlab + ``` + +1. Restart GitLab: + + ```sh + sudo service gitlab restart + ``` + +1. Verify that everything is configured correctly: + + ```sh + sudo -u git -H bundle exec rake gitlab:incoming_email:check RAILS_ENV=production + ``` + +1. Reply by email should now be working. + +### Development + +1. Go to the GitLab installation directory. + +1. Find the `incoming_email` section in `config/gitlab.yml`, enable the feature and fill in the details for your specific IMAP server and email account: + + ```yaml + # Configuration for Gmail / Google Apps, assumes mailbox gitlab-incoming@gmail.com + incoming_email: + enabled: true + + # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. + # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + address: "gitlab-incoming+%{key}@gmail.com" + + # Email account username + # With third party providers, this is usually the full email address. + # With self-hosted email servers, this is usually the user part of the email address. + user: "gitlab-incoming@gmail.com" + # Email account password + password: "[REDACTED]" + + # IMAP server host + host: "imap.gmail.com" + # IMAP server port + port: 993 + # Whether the IMAP server uses SSL + ssl: true + # Whether the IMAP server uses StartTLS + start_tls: false + + # The mailbox where incoming mail will end up. Usually "inbox". + mailbox: "inbox" + ``` + + As mentioned, the part after `+` is ignored, and this will end up in the mailbox for `gitlab-incoming@gmail.com`. + +1. Uncomment the `mail_room` line in your `Procfile`: + + ```yaml + mail_room: bundle exec mail_room -q -c config/mail_room.yml + ``` + +1. Restart GitLab: + + ```sh + bundle exec foreman start + ``` + +1. Verify that everything is configured correctly: + + ```sh + bundle exec rake gitlab:incoming_email:check RAILS_ENV=development + ``` + +1. Reply by email should now be working. diff --git a/doc/administration/reply_by_email_postfix_setup.md b/doc/administration/reply_by_email_postfix_setup.md new file mode 100644 index 00000000000..22f10489a6c --- /dev/null +++ b/doc/administration/reply_by_email_postfix_setup.md @@ -0,0 +1,324 @@ +# Set up Postfix for Reply by email + +This document will take you through the steps of setting up a basic Postfix mail +server with IMAP authentication on Ubuntu, to be used with [Reply by email]. + +The instructions make the assumption that you will be using the email address `incoming@gitlab.example.com`, that is, username `incoming` on host `gitlab.example.com`. Don't forget to change it to your actual host when executing the example code snippets. + +## Configure your server firewall + +1. Open up port 25 on your server so that people can send email into the server over SMTP. +2. If the mail server is different from the server running GitLab, open up port 143 on your server so that GitLab can read email from the server over IMAP. + +## Install packages + +1. Install the `postfix` package if it is not installed already: + + ```sh + sudo apt-get install postfix + ``` + + When asked about the environment, select 'Internet Site'. When asked to confirm the hostname, make sure it matches `gitlab.example.com`. + +1. Install the `mailutils` package. + + ```sh + sudo apt-get install mailutils + ``` + +## Create user + +1. Create a user for incoming email. + + ```sh + sudo useradd -m -s /bin/bash incoming + ``` + +1. Set a password for this user. + + ```sh + sudo passwd incoming + ``` + + Be sure not to forget this, you'll need it later. + +## Test the out-of-the-box setup + +1. Connect to the local SMTP server: + + ```sh + telnet localhost 25 + ``` + + You should see a prompt like this: + + ```sh + Trying 127.0.0.1... + Connected to localhost. + Escape character is '^]'. + 220 gitlab.example.com ESMTP Postfix (Ubuntu) + ``` + + If you get a `Connection refused` error instead, verify that `postfix` is running: + + ```sh + sudo postfix status + ``` + + If it is not, start it: + + ```sh + sudo postfix start + ``` + +1. Send the new `incoming` user a dummy email to test SMTP, by entering the following into the SMTP prompt: + + ``` + ehlo localhost + mail from: root@localhost + rcpt to: incoming@localhost + data + Subject: Re: Some issue + + Sounds good! + . + quit + ``` + + _**Note:** The `.` is a literal period on its own line._ + + _**Note:** If you receive an error after entering `rcpt to: incoming@localhost` + then your Postfix `my_network` configuration is not correct. The error will + say 'Temporary lookup failure'. See + [Configure Postfix to receive email from the Internet](#configure-postfix-to-receive-email-from-the-internet)._ + +1. Check if the `incoming` user received the email: + + ```sh + su - incoming + mail + ``` + + You should see output like this: + + ``` + "/var/mail/incoming": 1 message 1 unread + >U 1 root@localhost 59/2842 Re: Some issue + ``` + + Quit the mail app: + + ```sh + q + ``` + +1. Log out of the `incoming` account and go back to being `root`: + + ```sh + logout + ``` + +## Configure Postfix to use Maildir-style mailboxes + +Courier, which we will install later to add IMAP authentication, requires mailboxes to have the Maildir format, rather than mbox. + +1. Configure Postfix to use Maildir-style mailboxes: + + ```sh + sudo postconf -e "home_mailbox = Maildir/" + ``` + +1. Restart Postfix: + + ```sh + sudo /etc/init.d/postfix restart + ``` + +1. Test the new setup: + + 1. Follow steps 1 and 2 of _[Test the out-of-the-box setup](#test-the-out-of-the-box-setup)_. + 1. Check if the `incoming` user received the email: + + ```sh + su - incoming + MAIL=/home/incoming/Maildir + mail + ``` + + You should see output like this: + + ``` + "/home/incoming/Maildir": 1 message 1 unread + >U 1 root@localhost 59/2842 Re: Some issue + ``` + + Quit the mail app: + + ```sh + q + ``` + + _**Note:** If `mail` returns an error `Maildir: Is a directory` then your + version of `mail` doesn't support Maildir style mailboxes. Install + `heirloom-mailx` by running `sudo apt-get install heirloom-mailx`. Then, + try the above steps again, substituting `heirloom-mailx` for the `mail` + command._ + +1. Log out of the `incoming` account and go back to being `root`: + + ```sh + logout + ``` + +## Install the Courier IMAP server + +1. Install the `courier-imap` package: + + ```sh + sudo apt-get install courier-imap + ``` + +## Configure Postfix to receive email from the internet + +1. Let Postfix know about the domains that it should consider local: + + ```sh + sudo postconf -e "mydestination = gitlab.example.com, localhost.localdomain, localhost" + ``` + +1. Let Postfix know about the IPs that it should consider part of the LAN: + + We'll assume `192.168.1.0/24` is your local LAN. You can safely skip this step if you don't have other machines in the same local network. + + ```sh + sudo postconf -e "mynetworks = 127.0.0.0/8, 192.168.1.0/24" + ``` + +1. Configure Postfix to receive mail on all interfaces, which includes the internet: + + ```sh + sudo postconf -e "inet_interfaces = all" + ``` + +1. Configure Postfix to use the `+` delimiter for sub-addressing: + + ```sh + sudo postconf -e "recipient_delimiter = +" + ``` + +1. Restart Postfix: + + ```sh + sudo service postfix restart + ``` + +## Test the final setup + +1. Test SMTP under the new setup: + + 1. Connect to the SMTP server: + + ```sh + telnet gitlab.example.com 25 + ``` + + You should see a prompt like this: + + ```sh + Trying 123.123.123.123... + Connected to gitlab.example.com. + Escape character is '^]'. + 220 gitlab.example.com ESMTP Postfix (Ubuntu) + ``` + + If you get a `Connection refused` error instead, make sure your firewall is setup to allow inbound traffic on port 25. + + 1. Send the `incoming` user a dummy email to test SMTP, by entering the following into the SMTP prompt: + + ``` + ehlo gitlab.example.com + mail from: root@gitlab.example.com + rcpt to: incoming@gitlab.example.com + data + Subject: Re: Some issue + + Sounds good! + . + quit + ``` + + (Note: The `.` is a literal period on its own line) + + 1. Check if the `incoming` user received the email: + + ```sh + su - incoming + MAIL=/home/incoming/Maildir + mail + ``` + + You should see output like this: + + ``` + "/home/incoming/Maildir": 1 message 1 unread + >U 1 root@gitlab.example.com 59/2842 Re: Some issue + ``` + + Quit the mail app: + + ```sh + q + ``` + + 1. Log out of the `incoming` account and go back to being `root`: + + ```sh + logout + ``` + +1. Test IMAP under the new setup: + + 1. Connect to the IMAP server: + + ```sh + telnet gitlab.example.com 143 + ``` + + You should see a prompt like this: + + ```sh + Trying 123.123.123.123... + Connected to mail.example.gitlab.com. + Escape character is '^]'. + - OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION] Courier-IMAP ready. Copyright 1998-2011 Double Precision, Inc. See COPYING for distribution information. + ``` + + 1. Sign in as the `incoming` user to test IMAP, by entering the following into the IMAP prompt: + + ``` + a login incoming PASSWORD + ``` + + Replace PASSWORD with the password you set on the `incoming` user earlier. + + You should see output like this: + + ``` + a OK LOGIN Ok. + ``` + + 1. Disconnect from the IMAP server: + + ```sh + a logout + ``` + +## Done! + +If all the tests were successful, Postfix is all set up and ready to receive email! Continue with the [Reply by email](./README.md) guide to configure GitLab. + +--- + +_This document was adapted from https://help.ubuntu.com/community/PostfixBasicSetupHowto, by contributors to the Ubuntu documentation wiki._ + +[reply by email]: reply_by_email.md diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md index 483060395dd..b561c2f82aa 100644 --- a/doc/administration/restart_gitlab.md +++ b/doc/administration/restart_gitlab.md @@ -139,7 +139,7 @@ If you are using other init systems, like systemd, you can check the [omnibus-dl]: https://about.gitlab.com/downloads/ "Download the Omnibus packages" [install]: ../install/installation.md "Documentation to install GitLab from source" -[mailroom]: ../incoming_email/README.md "Used for replying by email in GitLab issues and merge requests" +[mailroom]: reply_by_email.md "Used for replying by email in GitLab issues and merge requests" [chef]: https://www.chef.io/chef/ "Chef official website" [src-service]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/support/init.d/gitlab "GitLab init service file" [gl-recipes]: https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/init "GitLab Recipes repository" diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index d127d7b85e5..d8dce4388e1 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -144,14 +144,14 @@ separate Rails process to debug the issue: 1. Obtain the private token for your user (Profile Settings -> Account). 1. Bring up the GitLab Rails console. For omnibus users, run: - ```` + ``` sudo gitlab-rails console ``` 1. At the Rails console, run: ```ruby - [1] pry(main)> app.get '<URL FROM STEP 1>/private_token?<TOKEN FROM STEP 2>' + [1] pry(main)> app.get '<URL FROM STEP 2>/?private_token=<TOKEN FROM STEP 3>' ``` For example: diff --git a/doc/api/README.md b/doc/api/README.md index 8e4f7f12b4b..3fbe5197a21 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -17,6 +17,8 @@ following locations: - [Commits](commits.md) - [Deployments](deployments.md) - [Deploy Keys](deploy_keys.md) +- [Gitignores templates](templates/gitignores.md) +- [GitLab CI Config templates](templates/gitlab_ci_ymls.md) - [Groups](groups.md) - [Group Access Requests](access_requests.md) - [Group Members](members.md) @@ -25,7 +27,7 @@ following locations: - [Labels](labels.md) - [Merge Requests](merge_requests.md) - [Milestones](milestones.md) -- [Open source license templates](licenses.md) +- [Open source license templates](templates/licenses.md) - [Namespaces](namespaces.md) - [Notes](notes.md) (comments) - [Notification settings](notification_settings.md) @@ -46,6 +48,7 @@ following locations: - [Todos](todos.md) - [Users](users.md) - [Validate CI configuration](ci/lint.md) +- [Version](version.md) ### Internal CI API @@ -101,7 +104,7 @@ Once you have your token, pass it to the API using either the `private_token` parameter or the `PRIVATE-TOKEN` header. -###Â Session cookie +### Session Cookie When signing in to GitLab as an ordinary user, a `_gitlab_session` cookie is set. The API will use this cookie for authentication if it is present, but using @@ -355,6 +358,19 @@ follows: } ``` +## Unknown route + +When you try to access an API URL that does not exist you will receive 404 Not Found. + +``` +HTTP/1.1 404 Not Found +Content-Type: application/json +{ + "error": "404 Not Found" +} +``` + + ## Clients There are many unofficial GitLab API Clients for most of the popular diff --git a/doc/api/award_emoji.md b/doc/api/award_emoji.md index c464e3f3f71..06111f4ab67 100644 --- a/doc/api/award_emoji.md +++ b/doc/api/award_emoji.md @@ -43,7 +43,7 @@ Example Response: "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "http://gitlab.example.com/u/root" + "web_url": "http://gitlab.example.com/root" }, "created_at": "2016-06-15T10:09:34.206Z", "updated_at": "2016-06-15T10:09:34.206Z", @@ -59,7 +59,7 @@ Example Response: "id": 26, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/7e65550957227bd38fe2d7fbc6fd2f7b?s=80&d=identicon", - "web_url": "http://gitlab.example.com/u/user4" + "web_url": "http://gitlab.example.com/user4" }, "created_at": "2016-06-15T10:09:34.177Z", "updated_at": "2016-06-15T10:09:34.177Z", @@ -103,7 +103,7 @@ Example Response: "id": 26, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/7e65550957227bd38fe2d7fbc6fd2f7b?s=80&d=identicon", - "web_url": "http://gitlab.example.com/u/user4" + "web_url": "http://gitlab.example.com/user4" }, "created_at": "2016-06-15T10:09:34.177Z", "updated_at": "2016-06-15T10:09:34.177Z", @@ -146,7 +146,7 @@ Example Response: "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "http://gitlab.example.com/u/root" + "web_url": "http://gitlab.example.com/root" }, "created_at": "2016-06-17T17:47:29.266Z", "updated_at": "2016-06-17T17:47:29.266Z", @@ -190,7 +190,7 @@ Example Response: "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "http://gitlab.example.com/u/root" + "web_url": "http://gitlab.example.com/root" }, "created_at": "2016-06-17T17:47:29.266Z", "updated_at": "2016-06-17T17:47:29.266Z", @@ -238,7 +238,7 @@ Example Response: "id": 26, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/7e65550957227bd38fe2d7fbc6fd2f7b?s=80&d=identicon", - "web_url": "http://gitlab.example.com/u/user4" + "web_url": "http://gitlab.example.com/user4" }, "created_at": "2016-06-15T10:09:34.197Z", "updated_at": "2016-06-15T10:09:34.197Z", @@ -279,7 +279,7 @@ Example Response: "id": 26, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/7e65550957227bd38fe2d7fbc6fd2f7b?s=80&d=identicon", - "web_url": "http://gitlab.example.com/u/user4" + "web_url": "http://gitlab.example.com/user4" }, "created_at": "2016-06-15T10:09:34.197Z", "updated_at": "2016-06-15T10:09:34.197Z", @@ -319,7 +319,7 @@ Example Response: "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "http://gitlab.example.com/u/root" + "web_url": "http://gitlab.example.com/root" }, "created_at": "2016-06-17T19:59:55.888Z", "updated_at": "2016-06-17T19:59:55.888Z", @@ -362,7 +362,7 @@ Example Response: "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "http://gitlab.example.com/u/root" + "web_url": "http://gitlab.example.com/root" }, "created_at": "2016-06-17T19:59:55.888Z", "updated_at": "2016-06-17T19:59:55.888Z", diff --git a/doc/api/boards.md b/doc/api/boards.md new file mode 100644 index 00000000000..28681719f43 --- /dev/null +++ b/doc/api/boards.md @@ -0,0 +1,251 @@ +# Boards + +Every API call to boards must be authenticated. + +If a user is not a member of a project and the project is private, a `GET` +request on that project will result to a `404` status code. + +## Project Board + +Lists Issue Boards in the given project. + +``` +GET /projects/:id/boards +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | The ID of a project | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/projects/:id/boards +``` + +Example response: + +```json +[ + { + "id" : 1, + "lists" : [ + { + "id" : 1, + "label" : { + "name" : "Testing", + "color" : "#F0AD4E", + "description" : null + }, + "position" : 1 + }, + { + "id" : 2, + "label" : { + "name" : "Ready", + "color" : "#FF0000", + "description" : null + }, + "position" : 2 + }, + { + "id" : 3, + "label" : { + "name" : "Production", + "color" : "#FF5F00", + "description" : null + }, + "position" : 3 + } + ] + } +] +``` + +## List board lists + +Get a list of the board's lists. +Does not include `backlog` and `done` lists + +``` +GET /projects/:id/boards/:board_id/lists +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | The ID of a project | +| `board_id` | integer | yes | The ID of a board | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/projects/5/boards/1/lists +``` + +Example response: + +```json +[ + { + "id" : 1, + "label" : { + "name" : "Testing", + "color" : "#F0AD4E", + "description" : null + }, + "position" : 1 + }, + { + "id" : 2, + "label" : { + "name" : "Ready", + "color" : "#FF0000", + "description" : null + }, + "position" : 2 + }, + { + "id" : 3, + "label" : { + "name" : "Production", + "color" : "#FF5F00", + "description" : null + }, + "position" : 3 + } +] +``` + +## Single board list + +Get a single board list. + +``` +GET /projects/:id/boards/:board_id/lists/:list_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | The ID of a project | +| `board_id` | integer | yes | The ID of a board | +| `list_id`| integer | yes | The ID of a board's list | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/projects/5/boards/1/lists/1 +``` + +Example response: + +```json +{ + "id" : 1, + "label" : { + "name" : "Testing", + "color" : "#F0AD4E", + "description" : null + }, + "position" : 1 +} +``` + +## New board list + +Creates a new Issue Board list. + +If the operation is successful, a status code of `200` and the newly-created +list is returned. If an error occurs, an error number and a message explaining +the reason is returned. + +``` +POST /projects/:id/boards/:board_id/lists +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | The ID of a project | +| `board_id` | integer | yes | The ID of a board | +| `label_id` | integer | yes | The ID of a label | + +```bash +curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/projects/5/boards/1/lists?label_id=5 +``` + +Example response: + +```json +{ + "id" : 1, + "label" : { + "name" : "Testing", + "color" : "#F0AD4E", + "description" : null + }, + "position" : 1 +} +``` + +## Edit board list + +Updates an existing Issue Board list. This call is used to change list position. + +If the operation is successful, a code of `200` and the updated board list is +returned. If an error occurs, an error number and a message explaining the +reason is returned. + +``` +PUT /projects/:id/boards/:board_id/lists/:list_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | The ID of a project | +| `board_id` | integer | yes | The ID of a board | +| `list_id` | integer | yes | The ID of a board's list | +| `position` | integer | yes | The position of the list | + +```bash +curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/projects/5/boards/1/lists/1?position=2 +``` + +Example response: + +```json +{ + "id" : 1, + "label" : { + "name" : "Testing", + "color" : "#F0AD4E", + "description" : null + }, + "position" : 1 +} +``` + +## Delete a board list + +Only for admins and project owners. Soft deletes the board list in question. +If the operation is successful, a status code `200` is returned. In case you cannot +destroy this board list, or it is not present, code `404` is given. + +``` +DELETE /projects/:id/boards/:board_id/lists/:list_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | The ID of a project | +| `board_id` | integer | yes | The ID of a board | +| `list_id` | integer | yes | The ID of a board's list | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/projects/5/boards/1/lists/1 +``` +Example response: + +```json +{ + "id" : 1, + "label" : { + "name" : "Testing", + "color" : "#F0AD4E", + "description" : null + }, + "position" : 1 +} +``` diff --git a/doc/api/builds.md b/doc/api/builds.md index dce666445d0..0476cac0eda 100644 --- a/doc/api/builds.md +++ b/doc/api/builds.md @@ -11,10 +11,10 @@ GET /projects/:id/builds | Attribute | Type | Required | Description | |-----------|---------|----------|---------------------| | `id` | integer | yes | The ID of a project | -| `scope` | string **or** array of strings | no | The scope of builds to show, one or array of: `pending`, `running`, `failed`, `success`, `canceled`; showing all builds if none provided | +| `scope` | string **or** array of strings | no | The scope of builds to show, one or array of: `created`, `pending`, `running`, `failed`, `success`, `canceled`, `skipped`; showing all builds if none provided | ``` -curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/projects/1/builds" +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" 'https://gitlab.example.com/api/v3/projects/1/builds?scope%5B0%5D=pending&scope%5B1%5D=running' ``` Example of response @@ -40,6 +40,12 @@ Example of response "finished_at": "2015-12-24T17:54:27.895Z", "id": 7, "name": "teaspoon", + "pipeline": { + "id": 6, + "ref": "master", + "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd", + "status": "pending" + } "ref": "master", "runner": null, "stage": "test", @@ -58,7 +64,7 @@ Example of response "state": "active", "twitter": "", "username": "root", - "web_url": "http://gitlab.dev/u/root", + "web_url": "http://gitlab.dev/root", "website_url": "" } }, @@ -78,6 +84,12 @@ Example of response "finished_at": "2015-12-24T17:54:24.921Z", "id": 6, "name": "spinach:other", + "pipeline": { + "id": 6, + "ref": "master", + "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd", + "status": "pending" + } "ref": "master", "runner": null, "stage": "test", @@ -96,7 +108,7 @@ Example of response "state": "active", "twitter": "", "username": "root", - "web_url": "http://gitlab.dev/u/root", + "web_url": "http://gitlab.dev/root", "website_url": "" } } @@ -120,10 +132,10 @@ GET /projects/:id/repository/commits/:sha/builds |-----------|---------|----------|---------------------| | `id` | integer | yes | The ID of a project | | `sha` | string | yes | The SHA id of a commit | -| `scope` | string **or** array of strings | no | The scope of builds to show, one or array of: `pending`, `running`, `failed`, `success`, `canceled`; showing all builds if none provided | +| `scope` | string **or** array of strings | no | The scope of builds to show, one or array of: `created`, `pending`, `running`, `failed`, `success`, `canceled`, `skipped`; showing all builds if none provided | ``` -curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/projects/1/repository/commits/0ff3ae198f8601a285adcf5c0fff204ee6fba5fd/builds" +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" 'https://gitlab.example.com/api/v3/projects/1/repository/commits/0ff3ae198f8601a285adcf5c0fff204ee6fba5fd/builds?scope%5B0%5D=pending&scope%5B1%5D=running' ``` Example of response @@ -146,6 +158,12 @@ Example of response "finished_at": "2016-01-11T10:14:09.526Z", "id": 69, "name": "rubocop", + "pipeline": { + "id": 6, + "ref": "master", + "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd", + "status": "pending" + } "ref": "master", "runner": null, "stage": "test", @@ -170,6 +188,12 @@ Example of response "finished_at": "2015-12-24T17:54:33.913Z", "id": 9, "name": "brakeman", + "pipeline": { + "id": 6, + "ref": "master", + "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd", + "status": "pending" + } "ref": "master", "runner": null, "stage": "test", @@ -188,7 +212,7 @@ Example of response "state": "active", "twitter": "", "username": "root", - "web_url": "http://gitlab.dev/u/root", + "web_url": "http://gitlab.dev/root", "website_url": "" } } @@ -231,6 +255,12 @@ Example of response "finished_at": "2015-12-24T17:54:31.198Z", "id": 8, "name": "rubocop", + "pipeline": { + "id": 6, + "ref": "master", + "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd", + "status": "pending" + } "ref": "master", "runner": null, "stage": "test", @@ -249,7 +279,7 @@ Example of response "state": "active", "twitter": "", "username": "root", - "web_url": "http://gitlab.dev/u/root", + "web_url": "http://gitlab.dev/root", "website_url": "" } } diff --git a/doc/api/ci/runners.md b/doc/api/ci/runners.md index ecec53fde03..16028d1f124 100644 --- a/doc/api/ci/runners.md +++ b/doc/api/ci/runners.md @@ -12,7 +12,9 @@ communication channel. For the consumer API see the This API uses two types of authentication: 1. Unique Runner's token, which is the token assigned to the Runner after it - has been registered. + has been registered. This token can be found on the Runner's edit page (go to + **Project > Runners**, select one of the Runners listed under **Runners activated for + this project**). 2. Using Runners' registration token. This is a token that can be found in project's settings. @@ -48,7 +50,7 @@ DELETE /ci/api/v1/runners/delete | Attribute | Type | Required | Description | | --------- | ------- | --------- | ----------- | -| `token` | string | yes | Runner's registration token | +| `token` | string | yes | Unique Runner's token | Example request: diff --git a/doc/api/commits.md b/doc/api/commits.md index 682151d4b1d..e1ed99d98d3 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -46,6 +46,91 @@ Example response: ] ``` +## Create a commit with multiple files and actions + +> [Introduced][ce-6096] in GitLab 8.13. + +Create a commit by posting a JSON payload + +``` +POST /projects/:id/repository/commits +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of a project or NAMESPACE/PROJECT_NAME | +| `branch_name` | string | yes | The name of a branch | +| `commit_message` | string | yes | Commit message | +| `actions[]` | array | yes | An array of action hashes to commit as a batch. See the next table for what attributes it can take. | +| `author_email` | string | no | Specify the commit author's email address | +| `author_name` | string | no | Specify the commit author's name | + + +| `actions[]` Attribute | Type | Required | Description | +| --------------------- | ---- | -------- | ----------- | +| `action` | string | yes | The action to perform, `create`, `delete`, `move`, `update` | +| `file_path` | string | yes | Full path to the file. Ex. `lib/class.rb` | +| `previous_path` | string | no | Original full path to the file being moved. Ex. `lib/class1.rb` | +| `content` | string | no | File content, required for all except `delete`. Optional for `move` | +| `encoding` | string | no | `text` or `base64`. `text` is default. | + +```bash +PAYLOAD=$(cat << 'JSON' +{ + "branch_name": "master", + "commit_message": "some commit message", + "actions": [ + { + "action": "create", + "file_path": "foo/bar", + "content": "some content" + }, + { + "action": "delete", + "file_path": "foo/bar2", + }, + { + "action": "move", + "file_path": "foo/bar3", + "previous_path": "foo/bar4", + "content": "some content" + }, + { + "action": "update", + "file_path": "foo/bar5", + "content": "new content" + } + ] +} +JSON +) +curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --header "Content-Type: application/json" --data "$PAYLOAD" https://gitlab.example.com/api/v3/projects/1/repository/commits +``` + +Example response: +```json +{ + "id": "ed899a2f4b50b4370feeea94676502b42383c746", + "short_id": "ed899a2f4b5", + "title": "some commit message", + "author_name": "Dmitriy Zaporozhets", + "author_email": "dzaporozhets@sphereconsultinginc.com", + "created_at": "2016-09-20T09:26:24.000-07:00", + "message": "some commit message", + "parent_ids": [ + "ae1d9fb46aa2b07ee9836d49862ec4e2c46fbbba" + ], + "committed_date": "2016-09-20T09:26:24.000-07:00", + "authored_date": "2016-09-20T09:26:24.000-07:00", + "stats": { + "additions": 2, + "deletions": 2, + "total": 4 + }, + "status": null +} +``` + ## Get a single commit Get a specific commit identified by the commit hash or name of a branch or tag. @@ -203,7 +288,7 @@ Example response: ```json { "author" : { - "web_url" : "https://gitlab.example.com/u/thedude", + "web_url" : "https://gitlab.example.com/thedude", "avatar_url" : "https://gitlab.example.com/uploads/user/avatar/28/The-Big-Lebowski-400-400.png", "username" : "thedude", "state" : "active", @@ -234,7 +319,7 @@ GET /projects/:id/repository/commits/:sha/statuses | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID of a project or NAMESPACE/PROJECT_NAME owned by the authenticated user | `sha` | string | yes | The commit SHA -| `ref_name`| string | no | The name of a repository branch or tag or, if not given, the default branch +| `ref` | string | no | The name of a repository branch or tag or, if not given, the default branch | `stage` | string | no | Filter by [build stage](../ci/yaml/README.md#stages), e.g., `test` | `name` | string | no | Filter by [job name](../ci/yaml/README.md#jobs), e.g., `bundler:audit` | `all` | boolean | no | Return all statuses, not only the latest ones @@ -258,7 +343,7 @@ Example response: "author" : { "username" : "thedude", "state" : "active", - "web_url" : "https://gitlab.example.com/u/thedude", + "web_url" : "https://gitlab.example.com/thedude", "avatar_url" : "https://gitlab.example.com/uploads/user/avatar/28/The-Big-Lebowski-400-400.png", "id" : 28, "name" : "Jeff Lebowski" @@ -285,7 +370,7 @@ Example response: "id" : 28, "name" : "Jeff Lebowski", "username" : "thedude", - "web_url" : "https://gitlab.example.com/u/thedude", + "web_url" : "https://gitlab.example.com/thedude", "state" : "active", "avatar_url" : "https://gitlab.example.com/uploads/user/avatar/28/The-Big-Lebowski-400-400.png" }, @@ -323,7 +408,7 @@ Example response: ```json { "author" : { - "web_url" : "https://gitlab.example.com/u/thedude", + "web_url" : "https://gitlab.example.com/thedude", "name" : "Jeff Lebowski", "avatar_url" : "https://gitlab.example.com/uploads/user/avatar/28/The-Big-Lebowski-400-400.png", "username" : "thedude", @@ -343,3 +428,5 @@ Example response: "finished_at" : "2016-01-19T09:05:50.365Z" } ``` + +[ce-6096]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6096 "Multi-file commit" diff --git a/doc/api/deployments.md b/doc/api/deployments.md index 417962de82d..3d95c4cde60 100644 --- a/doc/api/deployments.md +++ b/doc/api/deployments.md @@ -56,7 +56,7 @@ Example of response "state": "active", "twitter": "", "username": "root", - "web_url": "http://localhost:3000/u/root", + "web_url": "http://localhost:3000/root", "website_url": "" } }, @@ -75,7 +75,7 @@ Example of response "name": "Administrator", "state": "active", "username": "root", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" } }, { @@ -114,7 +114,7 @@ Example of response "state": "active", "twitter": "", "username": "root", - "web_url": "http://localhost:3000/u/root", + "web_url": "http://localhost:3000/root", "website_url": "" } }, @@ -133,7 +133,7 @@ Example of response "name": "Administrator", "state": "active", "username": "root", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" } } ] @@ -169,7 +169,7 @@ Example of response "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" }, "environment": { "id": 9, @@ -193,7 +193,7 @@ Example of response "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "http://localhost:3000/u/root", + "web_url": "http://localhost:3000/root", "created_at": "2016-08-11T07:09:20.351Z", "is_admin": true, "bio": null, diff --git a/doc/api/issues.md b/doc/api/issues.md index eed0d2fce51..134263d27b4 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -46,7 +46,7 @@ Example response: "author" : { "state" : "active", "id" : 18, - "web_url" : "https://gitlab.example.com/u/eileen.lowe", + "web_url" : "https://gitlab.example.com/eileen.lowe", "name" : "Alexandra Bashirian", "avatar_url" : null, "username" : "eileen.lowe" @@ -67,7 +67,7 @@ Example response: "state" : "active", "id" : 1, "name" : "Administrator", - "web_url" : "https://gitlab.example.com/u/root", + "web_url" : "https://gitlab.example.com/root", "avatar_url" : null, "username" : "root" }, @@ -134,7 +134,7 @@ Example response: }, "author" : { "state" : "active", - "web_url" : "https://gitlab.example.com/u/root", + "web_url" : "https://gitlab.example.com/root", "avatar_url" : null, "username" : "root", "id" : 1, @@ -145,7 +145,7 @@ Example response: "iid" : 1, "assignee" : { "avatar_url" : null, - "web_url" : "https://gitlab.example.com/u/lennie", + "web_url" : "https://gitlab.example.com/lennie", "state" : "active", "username" : "lennie", "id" : 9, @@ -215,7 +215,7 @@ Example response: }, "author" : { "state" : "active", - "web_url" : "https://gitlab.example.com/u/root", + "web_url" : "https://gitlab.example.com/root", "avatar_url" : null, "username" : "root", "id" : 1, @@ -226,7 +226,7 @@ Example response: "iid" : 1, "assignee" : { "avatar_url" : null, - "web_url" : "https://gitlab.example.com/u/lennie", + "web_url" : "https://gitlab.example.com/lennie", "state" : "active", "username" : "lennie", "id" : 9, @@ -281,7 +281,7 @@ Example response: }, "author" : { "state" : "active", - "web_url" : "https://gitlab.example.com/u/root", + "web_url" : "https://gitlab.example.com/root", "avatar_url" : null, "username" : "root", "id" : 1, @@ -292,7 +292,7 @@ Example response: "iid" : 1, "assignee" : { "avatar_url" : null, - "web_url" : "https://gitlab.example.com/u/lennie", + "web_url" : "https://gitlab.example.com/lennie", "state" : "active", "username" : "lennie", "id" : 9, @@ -357,7 +357,7 @@ Example response: "name" : "Alexandra Bashirian", "avatar_url" : null, "state" : "active", - "web_url" : "https://gitlab.example.com/u/eileen.lowe", + "web_url" : "https://gitlab.example.com/eileen.lowe", "id" : 18, "username" : "eileen.lowe" }, @@ -414,7 +414,7 @@ Example response: "username" : "eileen.lowe", "id" : 18, "state" : "active", - "web_url" : "https://gitlab.example.com/u/eileen.lowe" + "web_url" : "https://gitlab.example.com/eileen.lowe" }, "state" : "closed", "title" : "Issues with auth", @@ -500,7 +500,7 @@ Example response: "id": 12, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/46f6f7dc858ada7be1853f7fb96e81da?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/axel.block" + "web_url": "https://gitlab.example.com/axel.block" }, "author": { "name": "Kris Steuber", @@ -508,7 +508,7 @@ Example response: "id": 10, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/7a190fecbaa68212a4b68aeb6e3acd10?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/solon.cremin" + "web_url": "https://gitlab.example.com/solon.cremin" }, "due_date": null, "web_url": "http://example.com/example/example/issues/11", @@ -557,7 +557,7 @@ Example response: "id": 12, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/46f6f7dc858ada7be1853f7fb96e81da?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/axel.block" + "web_url": "https://gitlab.example.com/axel.block" }, "author": { "name": "Kris Steuber", @@ -565,7 +565,7 @@ Example response: "id": 10, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/7a190fecbaa68212a4b68aeb6e3acd10?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/solon.cremin" + "web_url": "https://gitlab.example.com/solon.cremin" }, "due_date": null, "web_url": "http://example.com/example/example/issues/11", @@ -614,7 +614,7 @@ Example response: "id": 21, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/3e6f06a86cf27fa8b56f3f74f7615987?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/keyon" + "web_url": "https://gitlab.example.com/keyon" }, "author": { "name": "Vivian Hermann", @@ -622,7 +622,7 @@ Example response: "id": 11, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/5224fd70153710e92fb8bcf79ac29d67?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/orville" + "web_url": "https://gitlab.example.com/orville" }, "subscribed": false, "due_date": null, @@ -669,7 +669,7 @@ Example response: "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/root" + "web_url": "https://gitlab.example.com/root" }, "action_name": "marked", "target_type": "Issue", @@ -700,7 +700,7 @@ Example response: "id": 14, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/a7fa515d53450023c83d62986d0658a8?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/francisca" + "web_url": "https://gitlab.example.com/francisca" }, "author": { "name": "Maxie Medhurst", @@ -708,7 +708,7 @@ Example response: "id": 12, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/a0d477b3ea21970ce6ffcbb817b0b435?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/craig_rutherford" + "web_url": "https://gitlab.example.com/craig_rutherford" }, "subscribed": true, "user_notes_count": 7, diff --git a/doc/api/keys.md b/doc/api/keys.md index faa6f212b43..b68f08a007d 100644 --- a/doc/api/keys.md +++ b/doc/api/keys.md @@ -24,7 +24,7 @@ Parameters: "id": 25, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/cfa35b8cd2ec278026357769582fa563?s=40\u0026d=identicon", - "web_url": "http://localhost:3000/u/john_smith", + "web_url": "http://localhost:3000/john_smith", "created_at": "2015-09-03T07:24:01.670Z", "is_admin": false, "bio": null, diff --git a/doc/api/labels.md b/doc/api/labels.md index 3653ccf304a..656232cc940 100644 --- a/doc/api/labels.md +++ b/doc/api/labels.md @@ -148,7 +148,7 @@ PUT /projects/:id/labels | --------------- | ------- | --------------------------------- | ------------------------------- | | `id` | integer | yes | The ID of the project | | `name` | string | yes | The name of the existing label | -| `new_name` | string | yes if `color` if not provided | The new name of the label | +| `new_name` | string | yes if `color` is not provided | The new name of the label | | `color` | string | yes if `new_name` is not provided | The new color of the label in 6-digit hex notation with leading `#` sign | | `description` | string | no | The new description of the label | diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index 494040a1ce8..f4167403c2c 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -621,7 +621,7 @@ Example response when the GitLab issue tracker is used: "author" : { "state" : "active", "id" : 18, - "web_url" : "https://gitlab.example.com/u/eileen.lowe", + "web_url" : "https://gitlab.example.com/eileen.lowe", "name" : "Alexandra Bashirian", "avatar_url" : null, "username" : "eileen.lowe" @@ -642,7 +642,7 @@ Example response when the GitLab issue tracker is used: "state" : "active", "id" : 1, "name" : "Administrator", - "web_url" : "https://gitlab.example.com/u/root", + "web_url" : "https://gitlab.example.com/root", "avatar_url" : null, "username" : "root" }, @@ -711,7 +711,7 @@ Example response: "id": 19, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/39ce4a2822cc896933ffbd68c1470e55?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/leila" + "web_url": "https://gitlab.example.com/leila" }, "assignee": { "name": "Celine Wehner", @@ -719,7 +719,7 @@ Example response: "id": 16, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/f4cd5605b769dd2ce405a27c6e6f2684?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/carli" + "web_url": "https://gitlab.example.com/carli" }, "source_project_id": 5, "target_project_id": 5, @@ -787,7 +787,7 @@ Example response: "id": 19, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/39ce4a2822cc896933ffbd68c1470e55?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/leila" + "web_url": "https://gitlab.example.com/leila" }, "assignee": { "name": "Celine Wehner", @@ -795,7 +795,7 @@ Example response: "id": 16, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/f4cd5605b769dd2ce405a27c6e6f2684?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/carli" + "web_url": "https://gitlab.example.com/carli" }, "source_project_id": 5, "target_project_id": 5, @@ -858,7 +858,7 @@ Example response: "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/root" + "web_url": "https://gitlab.example.com/root" }, "action_name": "marked", "target_type": "MergeRequest", @@ -881,7 +881,7 @@ Example response: "id": 14, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/a7fa515d53450023c83d62986d0658a8?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/francisca" + "web_url": "https://gitlab.example.com/francisca" }, "assignee": { "name": "Dr. Gabrielle Strosin", @@ -889,7 +889,7 @@ Example response: "id": 4, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/733005fcd7e6df12d2d8580171ccb966?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/barrett.krajcik" + "web_url": "https://gitlab.example.com/barrett.krajcik" }, "source_project_id": 3, "target_project_id": 3, diff --git a/doc/api/notes.md b/doc/api/notes.md index 572844b8b3f..58d40eecf3e 100644 --- a/doc/api/notes.md +++ b/doc/api/notes.md @@ -143,7 +143,7 @@ Example Response: "state": "active", "created_at": "2013-09-30T13:46:01Z", "avatar_url": "http://www.gravatar.com/avatar/5224fd70153710e92fb8bcf79ac29d67?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/pipin" + "web_url": "https://gitlab.example.com/pipin" }, "created_at": "2016-04-05T22:10:44.164Z", "system": false, @@ -268,7 +268,7 @@ Example Response: "state": "active", "created_at": "2013-09-30T13:46:01Z", "avatar_url": "http://www.gravatar.com/avatar/5224fd70153710e92fb8bcf79ac29d67?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/pipin" + "web_url": "https://gitlab.example.com/pipin" }, "created_at": "2016-04-06T16:51:53.239Z", "system": false, @@ -398,7 +398,7 @@ Example Response: "state": "active", "created_at": "2013-09-30T13:46:01Z", "avatar_url": "http://www.gravatar.com/avatar/5224fd70153710e92fb8bcf79ac29d67?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/pipin" + "web_url": "https://gitlab.example.com/pipin" }, "created_at": "2016-04-05T22:11:59.923Z", "system": false, diff --git a/doc/api/oauth2.md b/doc/api/oauth2.md index b34e8075775..5ef5e3f5744 100644 --- a/doc/api/oauth2.md +++ b/doc/api/oauth2.md @@ -1,4 +1,4 @@ -# GitLab as an OAuth2 client +# GitLab as an OAuth2 provider This document covers using the OAuth2 protocol to access GitLab. @@ -112,7 +112,7 @@ You can do POST request to `/oauth/token` with parameters: { "grant_type" : "password", "username" : "user@example.com", - "password" : "sekret" + "password" : "secret" } ``` @@ -130,8 +130,8 @@ For testing you can use the oauth2 ruby gem: ``` client = OAuth2::Client.new('the_client_id', 'the_client_secret', :site => "http://example.com") -access_token = client.password.get_token('user@example.com', 'sekret') +access_token = client.password.get_token('user@example.com', 'secret') puts access_token.token ``` -[personal access tokens]: ./README.md#personal-access-tokens +[personal access tokens]: ./README.md#personal-access-tokens
\ No newline at end of file diff --git a/doc/api/pipelines.md b/doc/api/pipelines.md index 847408a7f61..a29b3eb6f44 100644 --- a/doc/api/pipelines.md +++ b/doc/api/pipelines.md @@ -34,7 +34,7 @@ Example of response "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" }, "created_at": "2016-08-16T10:23:19.007Z", "updated_at": "2016-08-16T10:23:19.216Z", @@ -57,7 +57,7 @@ Example of response "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" }, "created_at": "2016-08-16T10:23:21.184Z", "updated_at": "2016-08-16T10:23:21.314Z", @@ -103,7 +103,7 @@ Example of response "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" }, "created_at": "2016-08-11T11:28:34.085Z", "updated_at": "2016-08-11T11:32:35.169Z", @@ -148,7 +148,7 @@ Response: "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" }, "created_at": "2016-08-11T11:28:34.085Z", "updated_at": "2016-08-11T11:32:35.169Z", @@ -193,7 +193,7 @@ Response: "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" }, "created_at": "2016-08-11T11:28:34.085Z", "updated_at": "2016-08-11T11:32:35.169Z", diff --git a/doc/api/projects.md b/doc/api/projects.md index 750ce1508df..b69db90e70d 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -20,7 +20,7 @@ Constants for project visibility levels are next: ## List projects -Get a list of projects accessible by the authenticated user. +Get a list of projects for which the authenticated user is a member. ``` GET /projects @@ -28,11 +28,14 @@ GET /projects Parameters: -- `archived` (optional) - if passed, limit by archived status -- `visibility` (optional) - if passed, limit by visibility `public`, `internal`, `private` -- `order_by` (optional) - Return requests ordered by `id`, `name`, `path`, `created_at`, `updated_at` or `last_activity_at` fields. Default is `created_at` -- `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` -- `search` (optional) - Return list of authorized projects according to a search criteria +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `archived` | boolean | no | Limit by archived status | +| `visibility` | string | no | Limit by visibility `public`, `internal`, or `private` | +| `order_by` | string | no | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at` | +| `sort` | string | no | Return projects sorted in `asc` or `desc` order. Default is `desc` | +| `search` | string | no | Return list of authorized projects matching the search criteria | +| `simple` | boolean | no | Return only the ID, URL, name, and path of each project | ```json [ @@ -153,6 +156,138 @@ Parameters: ] ``` +Get a list of projects which the authenticated user can see. + +``` +GET /projects/visible +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `archived` | boolean | no | Limit by archived status | +| `visibility` | string | no | Limit by visibility `public`, `internal`, or `private` | +| `order_by` | string | no | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at` | +| `sort` | string | no | Return projects sorted in `asc` or `desc` order. Default is `desc` | +| `search` | string | no | Return list of authorized projects matching the search criteria | +| `simple` | boolean | no | Return only the ID, URL, name, and path of each project | + +```json +[ + { + "id": 4, + "description": null, + "default_branch": "master", + "public": false, + "visibility_level": 0, + "ssh_url_to_repo": "git@example.com:diaspora/diaspora-client.git", + "http_url_to_repo": "http://example.com/diaspora/diaspora-client.git", + "web_url": "http://example.com/diaspora/diaspora-client", + "tag_list": [ + "example", + "disapora client" + ], + "owner": { + "id": 3, + "name": "Diaspora", + "created_at": "2013-09-30T13:46:02Z" + }, + "name": "Diaspora Client", + "name_with_namespace": "Diaspora / Diaspora Client", + "path": "diaspora-client", + "path_with_namespace": "diaspora/diaspora-client", + "issues_enabled": true, + "open_issues_count": 1, + "merge_requests_enabled": true, + "builds_enabled": true, + "wiki_enabled": true, + "snippets_enabled": false, + "container_registry_enabled": false, + "created_at": "2013-09-30T13:46:02Z", + "last_activity_at": "2013-09-30T13:46:02Z", + "creator_id": 3, + "namespace": { + "created_at": "2013-09-30T13:46:02Z", + "description": "", + "id": 3, + "name": "Diaspora", + "owner_id": 1, + "path": "diaspora", + "updated_at": "2013-09-30T13:46:02Z" + }, + "archived": false, + "avatar_url": "http://example.com/uploads/project/avatar/4/uploads/avatar.png", + "shared_runners_enabled": true, + "forks_count": 0, + "star_count": 0, + "runners_token": "b8547b1dc37721d05889db52fa2f02", + "public_builds": true, + "shared_with_groups": [] + }, + { + "id": 6, + "description": null, + "default_branch": "master", + "public": false, + "visibility_level": 0, + "ssh_url_to_repo": "git@example.com:brightbox/puppet.git", + "http_url_to_repo": "http://example.com/brightbox/puppet.git", + "web_url": "http://example.com/brightbox/puppet", + "tag_list": [ + "example", + "puppet" + ], + "owner": { + "id": 4, + "name": "Brightbox", + "created_at": "2013-09-30T13:46:02Z" + }, + "name": "Puppet", + "name_with_namespace": "Brightbox / Puppet", + "path": "puppet", + "path_with_namespace": "brightbox/puppet", + "issues_enabled": true, + "open_issues_count": 1, + "merge_requests_enabled": true, + "builds_enabled": true, + "wiki_enabled": true, + "snippets_enabled": false, + "container_registry_enabled": false, + "created_at": "2013-09-30T13:46:02Z", + "last_activity_at": "2013-09-30T13:46:02Z", + "creator_id": 3, + "namespace": { + "created_at": "2013-09-30T13:46:02Z", + "description": "", + "id": 4, + "name": "Brightbox", + "owner_id": 1, + "path": "brightbox", + "updated_at": "2013-09-30T13:46:02Z" + }, + "permissions": { + "project_access": { + "access_level": 10, + "notification_level": 3 + }, + "group_access": { + "access_level": 50, + "notification_level": 3 + } + }, + "archived": false, + "avatar_url": null, + "shared_runners_enabled": true, + "forks_count": 0, + "star_count": 0, + "runners_token": "b8547b1dc37721d05889db52fa2f02", + "public_builds": true, + "shared_with_groups": [] + } +] +``` + ### List owned projects Get a list of projects which are owned by the authenticated user. @@ -163,11 +298,13 @@ GET /projects/owned Parameters: -- `archived` (optional) - if passed, limit by archived status -- `visibility` (optional) - if passed, limit by visibility `public`, `internal`, `private` -- `order_by` (optional) - Return requests ordered by `id`, `name`, `path`, `created_at`, `updated_at` or `last_activity_at` fields. Default is `created_at` -- `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` -- `search` (optional) - Return list of authorized projects according to a search criteria +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `archived` | boolean | no | Limit by archived status | +| `visibility` | string | no | Limit by visibility `public`, `internal`, or `private` | +| `order_by` | string | no | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at` | +| `sort` | string | no | Return projects sorted in `asc` or `desc` order. Default is `desc` | +| `search` | string | no | Return list of authorized projects matching the search criteria | ### List starred projects @@ -179,11 +316,13 @@ GET /projects/starred Parameters: -- `archived` (optional) - if passed, limit by archived status -- `visibility` (optional) - if passed, limit by visibility `public`, `internal`, `private` -- `order_by` (optional) - Return requests ordered by `id`, `name`, `path`, `created_at`, `updated_at` or `last_activity_at` fields. Default is `created_at` -- `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` -- `search` (optional) - Return list of authorized projects according to a search criteria +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `archived` | boolean | no | Limit by archived status | +| `visibility` | string | no | Limit by visibility `public`, `internal`, or `private` | +| `order_by` | string | no | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at` | +| `sort` | string | no | Return projects sorted in `asc` or `desc` order. Default is `desc` | +| `search` | string | no | Return list of authorized projects matching the search criteria | ### List ALL projects @@ -195,11 +334,13 @@ GET /projects/all Parameters: -- `archived` (optional) - if passed, limit by archived status -- `visibility` (optional) - if passed, limit by visibility `public`, `internal`, `private` -- `order_by` (optional) - Return requests ordered by `id`, `name`, `path`, `created_at`, `updated_at` or `last_activity_at` fields. Default is `created_at` -- `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` -- `search` (optional) - Return list of authorized projects according to a search criteria +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `archived` | boolean | no | Limit by archived status | +| `visibility` | string | no | Limit by visibility `public`, `internal`, or `private` | +| `order_by` | string | no | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at` | +| `sort` | string | no | Return projects sorted in `asc` or `desc` order. Default is `desc` | +| `search` | string | no | Return list of authorized projects matching the search criteria | ### Get single project @@ -212,7 +353,9 @@ GET /projects/:id Parameters: -- `id` (required) - The ID or NAMESPACE/PROJECT_NAME of a project +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or NAMESPACE/PROJECT_NAME of the project | ```json { @@ -293,7 +436,7 @@ Parameters: ### Get project events Get the events for the specified project. -Sorted from newest to latest +Sorted from newest to oldest ``` GET /projects/:id/events @@ -301,7 +444,9 @@ GET /projects/:id/events Parameters: -- `id` (required) - The ID or NAMESPACE/PROJECT_NAME of a project +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or NAMESPACE/PROJECT_NAME of the project | ```json [ @@ -320,7 +465,7 @@ Parameters: "id": 1, "state": "active", "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" }, "author_username": "root" }, @@ -337,7 +482,7 @@ Parameters: "id": 1, "state": "active", "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" }, "author_username": "john", "data": { @@ -383,7 +528,7 @@ Parameters: "id": 1, "state": "active", "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" }, "author_username": "root" }, @@ -407,7 +552,7 @@ Parameters: "id": 1, "state": "active", "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" }, "created_at": "2015-12-04T10:33:56.698Z", "system": false, @@ -422,7 +567,7 @@ Parameters: "id": 1, "state": "active", "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", - "web_url": "http://localhost:3000/u/root" + "web_url": "http://localhost:3000/root" }, "author_username": "root" } @@ -439,24 +584,26 @@ POST /projects Parameters: -- `name` (required) - new project name -- `path` (optional) - custom repository name for new project. By default generated based on name -- `namespace_id` (optional) - namespace for the new project (defaults to user) -- `description` (optional) - short project description -- `issues_enabled` (optional) -- `merge_requests_enabled` (optional) -- `builds_enabled` (optional) -- `wiki_enabled` (optional) -- `snippets_enabled` (optional) -- `container_registry_enabled` (optional) -- `shared_runners_enabled` (optional) -- `public` (optional) - if `true` same as setting visibility_level = 20 -- `visibility_level` (optional) -- `import_url` (optional) -- `public_builds` (optional) -- `only_allow_merge_if_build_succeeds` (optional) -- `lfs_enabled` (optional) -- `request_access_enabled` (optional) - Allow users to request member access. +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `name` | string | yes | The name of the new project | +| `path` | string | no | Custom repository name for new project. By default generated based on name | +| `namespace_id` | integer | no | Namespace for the new project (defaults to the current user's namespace) | +| `description` | string | no | Short project description | +| `issues_enabled` | boolean | no | Enable issues for this project | +| `merge_requests_enabled` | boolean | no | Enable merge requests for this project | +| `builds_enabled` | boolean | no | Enable builds for this project | +| `wiki_enabled` | boolean | no | Enable wiki for this project | +| `snippets_enabled` | boolean | no | Enable snippets for this project | +| `container_registry_enabled` | boolean | no | Enable container registry for this project | +| `shared_runners_enabled` | boolean | no | Enable shared runners for this project | +| `public` | boolean | no | If `true`, the same as setting `visibility_level` to 20 | +| `visibility_level` | integer | no | See [project visibility level][#project-visibility-level] | +| `import_url` | string | no | URL to import repository from | +| `public_builds` | boolean | no | If `true`, builds can be viewed by non-project-members | +| `only_allow_merge_if_build_succeeds` | boolean | no | Set whether merge requests can only be merged with successful builds | +| `lfs_enabled` | boolean | no | Enable LFS | +| `request_access_enabled` | boolean | no | Allow users to request member access | ### Create project for user @@ -468,23 +615,27 @@ POST /projects/user/:user_id Parameters: -- `user_id` (required) - user_id of owner -- `name` (required) - new project name -- `description` (optional) - short project description -- `issues_enabled` (optional) -- `merge_requests_enabled` (optional) -- `builds_enabled` (optional) -- `wiki_enabled` (optional) -- `snippets_enabled` (optional) -- `container_registry_enabled` (optional) -- `shared_runners_enabled` (optional) -- `public` (optional) - if `true` same as setting visibility_level = 20 -- `visibility_level` (optional) -- `import_url` (optional) -- `public_builds` (optional) -- `only_allow_merge_if_build_succeeds` (optional) -- `lfs_enabled` (optional) -- `request_access_enabled` (optional) - Allow users to request member access. +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `user_id` | integer | yes | The user ID of the project owner | +| `name` | string | yes | The name of the new project | +| `path` | string | no | Custom repository name for new project. By default generated based on name | +| `namespace_id` | integer | no | Namespace for the new project (defaults to the current user's namespace) | +| `description` | string | no | Short project description | +| `issues_enabled` | boolean | no | Enable issues for this project | +| `merge_requests_enabled` | boolean | no | Enable merge requests for this project | +| `builds_enabled` | boolean | no | Enable builds for this project | +| `wiki_enabled` | boolean | no | Enable wiki for this project | +| `snippets_enabled` | boolean | no | Enable snippets for this project | +| `container_registry_enabled` | boolean | no | Enable container registry for this project | +| `shared_runners_enabled` | boolean | no | Enable shared runners for this project | +| `public` | boolean | no | If `true`, the same as setting `visibility_level` to 20 | +| `visibility_level` | integer | no | See [project visibility level][#project-visibility-level] | +| `import_url` | string | no | URL to import repository from | +| `public_builds` | boolean | no | If `true`, builds can be viewed by non-project-members | +| `only_allow_merge_if_build_succeeds` | boolean | no | Set whether merge requests can only be merged with successful builds | +| `lfs_enabled` | boolean | no | Enable LFS | +| `request_access_enabled` | boolean | no | Allow users to request member access | ### Edit project @@ -496,24 +647,26 @@ PUT /projects/:id Parameters: -- `id` (required) - The ID or NAMESPACE/PROJECT_NAME of a project -- `name` (optional) - project name -- `path` (optional) - repository name for project -- `description` (optional) - short project description -- `default_branch` (optional) -- `issues_enabled` (optional) -- `merge_requests_enabled` (optional) -- `builds_enabled` (optional) -- `wiki_enabled` (optional) -- `snippets_enabled` (optional) -- `container_registry_enabled` (optional) -- `shared_runners_enabled` (optional) -- `public` (optional) - if `true` same as setting visibility_level = 20 -- `visibility_level` (optional) -- `public_builds` (optional) -- `only_allow_merge_if_build_succeeds` (optional) -- `lfs_enabled` (optional) -- `request_access_enabled` (optional) - Allow users to request member access. +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or NAMESPACE/PROJECT_NAME of the project | +| `name` | string | yes | The name of the project | +| `path` | string | no | Custom repository name for the project. By default generated based on name | +| `description` | string | no | Short project description | +| `issues_enabled` | boolean | no | Enable issues for this project | +| `merge_requests_enabled` | boolean | no | Enable merge requests for this project | +| `builds_enabled` | boolean | no | Enable builds for this project | +| `wiki_enabled` | boolean | no | Enable wiki for this project | +| `snippets_enabled` | boolean | no | Enable snippets for this project | +| `container_registry_enabled` | boolean | no | Enable container registry for this project | +| `shared_runners_enabled` | boolean | no | Enable shared runners for this project | +| `public` | boolean | no | If `true`, the same as setting `visibility_level` to 20 | +| `visibility_level` | integer | no | See [project visibility level][#project-visibility-level] | +| `import_url` | string | no | URL to import repository from | +| `public_builds` | boolean | no | If `true`, builds can be viewed by non-project-members | +| `only_allow_merge_if_build_succeeds` | boolean | no | Set whether merge requests can only be merged with successful builds | +| `lfs_enabled` | boolean | no | Enable LFS | +| `request_access_enabled` | boolean | no | Allow users to request member access | On success, method returns 200 with the updated project. If parameters are invalid, 400 is returned. @@ -528,8 +681,10 @@ POST /projects/fork/:id Parameters: -- `id` (required) - The ID or NAMESPACE/PROJECT_NAME of the project to be forked -- `namespace` (optional) - The ID or path of the namespace that the project will be forked to +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or NAMESPACE/PROJECT_NAME of the project | +| `namespace` | integer/string | yes | The ID or path of the namespace that the project will be forked to | ### Star a project @@ -540,9 +695,11 @@ Stars a given project. Returns status code `201` and the project on success and POST /projects/:id/star ``` +Parameters: + | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | +| `id` | integer/string | yes | The ID or NAMESPACE/PROJECT_NAME of the project | ```bash curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/projects/5/star" @@ -610,7 +767,7 @@ DELETE /projects/:id/star | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | ```bash curl --request DELETE --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/projects/5/star" @@ -682,7 +839,7 @@ POST /projects/:id/archive | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | ```bash curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/projects/archive" @@ -770,7 +927,7 @@ POST /projects/:id/unarchive | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | ```bash curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/projects/unarchive" @@ -853,7 +1010,9 @@ DELETE /projects/:id Parameters: -- `id` (required) - The ID or NAMESPACE/PROJECT_NAME of the project to be forked +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | ## Uploads @@ -867,8 +1026,10 @@ POST /projects/:id/uploads Parameters: -- `id` (required) - The ID or NAMESPACE/PROJECT_NAME of the project to be forked -- `file` (required) - The file to be uploaded +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | +| `file` | string | yes | The file to be uploaded | ```json { @@ -896,9 +1057,12 @@ POST /projects/:id/share Parameters: -- `id` (required) - The ID or NAMESPACE/PROJECT_NAME of the project to be forked -- `group_id` (required) - The ID of a group -- `group_access` (required) - Level of permissions for sharing +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | +| `group_id` | integer | yes | The ID of the group to share with | +| `group_access` | integer | yes | The permissions level to grant the group | +| `expires_at` | string | no | Share expiration date in ISO 8601 format: 2016-09-26 | ## Hooks @@ -915,7 +1079,9 @@ GET /projects/:id/hooks Parameters: -- `id` (required) - The ID or NAMESPACE/PROJECT_NAME of a project +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | ### Get project hook @@ -927,8 +1093,10 @@ GET /projects/:id/hooks/:hook_id Parameters: -- `id` (required) - The ID or NAMESPACE/PROJECT_NAME of a project -- `hook_id` (required) - The ID of a project hook +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | +| `hook_id` | integer | yes | The ID of a project hook | ```json { @@ -958,17 +1126,19 @@ POST /projects/:id/hooks Parameters: -- `id` (required) - The ID or NAMESPACE/PROJECT_NAME of a project -- `url` (required) - The hook URL -- `push_events` - Trigger hook on push events -- `issues_events` - Trigger hook on issues events -- `merge_requests_events` - Trigger hook on merge_requests events -- `tag_push_events` - Trigger hook on push_tag events -- `note_events` - Trigger hook on note events -- `build_events` - Trigger hook on build events -- `pipeline_events` - Trigger hook on pipeline events -- `wiki_page_events` - Trigger hook on wiki page events -- `enable_ssl_verification` - Do SSL verification when triggering the hook +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | +| `url` | string | yes | The hook URL | +| `push_events` | boolean | no | Trigger hook on push events | +| `issues_events` | boolean | no | Trigger hook on issues events | +| `merge_requests_events` | boolean | no | Trigger hook on merge requests events | +| `tag_push_events` | boolean | no | Trigger hook on tag push events | +| `note_events` | boolean | no | Trigger hook on note events | +| `build_events` | boolean | no | Trigger hook on build events | +| `pipeline_events` | boolean | no | Trigger hook on pipeline events | +| `wiki_events` | boolean | no | Trigger hook on wiki events | +| `enable_ssl_verification` | boolean | no | Do SSL verification when triggering the hook | ### Edit project hook @@ -980,18 +1150,20 @@ PUT /projects/:id/hooks/:hook_id Parameters: -- `id` (required) - The ID or NAMESPACE/PROJECT_NAME of a project -- `hook_id` (required) - The ID of a project hook -- `url` (required) - The hook URL -- `push_events` - Trigger hook on push events -- `issues_events` - Trigger hook on issues events -- `merge_requests_events` - Trigger hook on merge_requests events -- `tag_push_events` - Trigger hook on push_tag events -- `note_events` - Trigger hook on note events -- `build_events` - Trigger hook on build events -- `pipeline_events` - Trigger hook on pipeline events -- `wiki_page_events` - Trigger hook on wiki page events -- `enable_ssl_verification` - Do SSL verification when triggering the hook +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | +| `hook_id` | integer | yes | The ID of the project hook | +| `url` | string | yes | The hook URL | +| `push_events` | boolean | no | Trigger hook on push events | +| `issues_events` | boolean | no | Trigger hook on issues events | +| `merge_requests_events` | boolean | no | Trigger hook on merge requests events | +| `tag_push_events` | boolean | no | Trigger hook on tag push events | +| `note_events` | boolean | no | Trigger hook on note events | +| `build_events` | boolean | no | Trigger hook on build events | +| `pipeline_events` | boolean | no | Trigger hook on pipeline events | +| `wiki_events` | boolean | no | Trigger hook on wiki events | +| `enable_ssl_verification` | boolean | no | Do SSL verification when triggering the hook | ### Delete project hook @@ -1004,8 +1176,10 @@ DELETE /projects/:id/hooks/:hook_id Parameters: -- `id` (required) - The ID or NAMESPACE/PROJECT_NAME of a project -- `hook_id` (required) - The ID of hook to delete +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | +| `hook_id` | integer | yes | The ID of the project hook | Note the JSON response differs if the hook is available or not. If the project hook is available before it is returned in the JSON response or an empty response is returned. @@ -1024,7 +1198,9 @@ GET /projects/:id/repository/branches Parameters: -- `id` (required) - The ID or NAMESPACE/PROJECT_NAME of a project +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | ```json [ @@ -1079,10 +1255,12 @@ GET /projects/:id/repository/branches/:branch Parameters: -- `id` (required) - The ID or NAMESPACE/PROJECT_NAME of a project -- `branch` (required) - The name of the branch. -- `developers_can_push` - Flag if developers can push to the branch. -- `developers_can_merge` - Flag if developers can merge to the branch. +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | +| `branch` | string | yes | The name of the branch | +| `developers_can_push` | boolean | no | Flag if developers can push to the branch | +| `developers_can_merge` | boolean | no | Flag if developers can merge to the branch | ### Protect single branch @@ -1094,8 +1272,10 @@ PUT /projects/:id/repository/branches/:branch/protect Parameters: -- `id` (required) - The ID or NAMESPACE/PROJECT_NAME of a project -- `branch` (required) - The name of the branch. +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | +| `branch` | string | yes | The name of the branch | ### Unprotect single branch @@ -1107,8 +1287,10 @@ PUT /projects/:id/repository/branches/:branch/unprotect Parameters: -- `id` (required) - The ID or NAMESPACE/PROJECT_NAME of a project -- `branch` (required) - The name of the branch. +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | +| `branch` | string | yes | The name of the branch | ## Admin fork relation @@ -1122,8 +1304,10 @@ POST /projects/:id/fork/:forked_from_id Parameters: -- `id` (required) - The ID or NAMESPACE/PROJECT_NAME of the project to be forked -- `forked_from_id:` (required) - The ID of the project that was forked from +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | +| `forked_from_id` | ID | yes | The ID of the project that was forked from | ### Delete an existing forked from relationship @@ -1133,7 +1317,9 @@ DELETE /projects/:id/fork Parameter: -- `id` (required) - The ID or NAMESPACE/PROJECT_NAME of the project to be forked +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | ## Search for projects by name @@ -1145,8 +1331,8 @@ GET /projects/search/:query Parameters: -- `query` (required) - A string contained in the project name -- `per_page` (optional) - number of projects to return per page -- `page` (optional) - the page to retrieve -- `order_by` (optional) - Return requests ordered by `id`, `name`, `created_at` or `last_activity_at` fields -- `sort` (optional) - Return requests sorted in `asc` or `desc` order +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `query` | string | yes | A string contained in the project name | +| `order_by` | string | no | Return requests ordered by `id`, `name`, `created_at` or `last_activity_at` fields | +| `sort` | string | no | Return requests sorted in `asc` or `desc` order | diff --git a/doc/api/settings.md b/doc/api/settings.md index aaa2c99642b..f7ad3b4cc8e 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -41,7 +41,9 @@ Example response: "gravatar_enabled" : true, "sign_in_text" : null, "container_registry_token_expire_delay": 5, - "repository_storage": "default" + "repository_storage": "default", + "koding_enabled": false, + "koding_url": null } ``` @@ -72,7 +74,9 @@ PUT /application/settings | `after_sign_out_path` | string | no | Where to redirect users after logout | | `container_registry_token_expire_delay` | integer | no | Container Registry token duration in minutes | | `repository_storage` | string | no | Storage path for new projects. The value should be the name of one of the repository storage paths defined in your gitlab.yml | -| `enabled_git_access_protocol` | string | no | Enabled protocols for Git access. Allowed values are: `ssh`, `http`, and `nil` to allow both protocols. +| `enabled_git_access_protocol` | string | no | Enabled protocols for Git access. Allowed values are: `ssh`, `http`, and `nil` to allow both protocols. | +| `koding_enabled` | boolean | no | Enable Koding integration. Default is `false`. | +| `koding_url` | string | yes (if `koding_enabled` is `true`) | The Koding instance URL for integration. | ```bash curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/application/settings?signup_enabled=false&default_project_visibility=1 @@ -103,6 +107,8 @@ Example response: "user_oauth_applications": true, "after_sign_out_path": "", "container_registry_token_expire_delay": 5, - "repository_storage": "default" + "repository_storage": "default", + "koding_enabled": false, + "koding_url": null } ``` diff --git a/doc/api/system_hooks.md b/doc/api/system_hooks.md index 1802fae14fe..073e99b7147 100644 --- a/doc/api/system_hooks.md +++ b/doc/api/system_hooks.md @@ -98,11 +98,8 @@ Example response: ## Delete system hook -Deletes a system hook. This is an idempotent API function and returns `200 OK` -even if the hook is not available. - -If the hook is deleted, a JSON object is returned. An error is raised if the -hook is not found. +Deletes a system hook. It returns `200 OK` if the hooks is deleted and +`404 Not Found` if the hook is not found. --- diff --git a/doc/api/templates/gitignores.md b/doc/api/templates/gitignores.md new file mode 100644 index 00000000000..8235be92b12 --- /dev/null +++ b/doc/api/templates/gitignores.md @@ -0,0 +1,579 @@ +# Gitignores + +## List gitignore templates + +Get all gitignore templates. + +``` +GET /templates/gitignores +``` + +```bash +curl https://gitlab.example.com/api/v3/templates/gitignores +``` + +Example response: + +```json +[ + { + "name": "AppEngine" + }, + { + "name": "Laravel" + }, + { + "name": "Elisp" + }, + { + "name": "SketchUp" + }, + { + "name": "Ada" + }, + { + "name": "Ruby" + }, + { + "name": "Kohana" + }, + { + "name": "Nanoc" + }, + { + "name": "Erlang" + }, + { + "name": "OCaml" + }, + { + "name": "Lithium" + }, + { + "name": "Fortran" + }, + { + "name": "Scala" + }, + { + "name": "Node" + }, + { + "name": "Fancy" + }, + { + "name": "Perl" + }, + { + "name": "Zephir" + }, + { + "name": "WordPress" + }, + { + "name": "Symfony" + }, + { + "name": "FuelPHP" + }, + { + "name": "DM" + }, + { + "name": "Sdcc" + }, + { + "name": "Rust" + }, + { + "name": "C" + }, + { + "name": "Umbraco" + }, + { + "name": "Actionscript" + }, + { + "name": "Android" + }, + { + "name": "Grails" + }, + { + "name": "Composer" + }, + { + "name": "ExpressionEngine" + }, + { + "name": "Gcov" + }, + { + "name": "Qt" + }, + { + "name": "Phalcon" + }, + { + "name": "ArchLinuxPackages" + }, + { + "name": "TeX" + }, + { + "name": "SCons" + }, + { + "name": "Lilypond" + }, + { + "name": "CommonLisp" + }, + { + "name": "Rails" + }, + { + "name": "Mercury" + }, + { + "name": "Magento" + }, + { + "name": "ChefCookbook" + }, + { + "name": "GitBook" + }, + { + "name": "C++" + }, + { + "name": "Eagle" + }, + { + "name": "Go" + }, + { + "name": "OpenCart" + }, + { + "name": "Scheme" + }, + { + "name": "Typo3" + }, + { + "name": "SeamGen" + }, + { + "name": "Swift" + }, + { + "name": "Elm" + }, + { + "name": "Unity" + }, + { + "name": "Agda" + }, + { + "name": "CUDA" + }, + { + "name": "VVVV" + }, + { + "name": "Finale" + }, + { + "name": "LemonStand" + }, + { + "name": "Textpattern" + }, + { + "name": "Julia" + }, + { + "name": "Packer" + }, + { + "name": "Scrivener" + }, + { + "name": "Dart" + }, + { + "name": "Plone" + }, + { + "name": "Jekyll" + }, + { + "name": "Xojo" + }, + { + "name": "LabVIEW" + }, + { + "name": "Autotools" + }, + { + "name": "KiCad" + }, + { + "name": "Prestashop" + }, + { + "name": "ROS" + }, + { + "name": "Smalltalk" + }, + { + "name": "GWT" + }, + { + "name": "OracleForms" + }, + { + "name": "SugarCRM" + }, + { + "name": "Nim" + }, + { + "name": "SymphonyCMS" + }, + { + "name": "Maven" + }, + { + "name": "CFWheels" + }, + { + "name": "Python" + }, + { + "name": "ZendFramework" + }, + { + "name": "CakePHP" + }, + { + "name": "Concrete5" + }, + { + "name": "PlayFramework" + }, + { + "name": "Terraform" + }, + { + "name": "Elixir" + }, + { + "name": "CMake" + }, + { + "name": "Joomla" + }, + { + "name": "Coq" + }, + { + "name": "Delphi" + }, + { + "name": "Haskell" + }, + { + "name": "Yii" + }, + { + "name": "Java" + }, + { + "name": "UnrealEngine" + }, + { + "name": "AppceleratorTitanium" + }, + { + "name": "CraftCMS" + }, + { + "name": "ForceDotCom" + }, + { + "name": "ExtJs" + }, + { + "name": "MetaProgrammingSystem" + }, + { + "name": "D" + }, + { + "name": "Objective-C" + }, + { + "name": "RhodesRhomobile" + }, + { + "name": "R" + }, + { + "name": "EPiServer" + }, + { + "name": "Yeoman" + }, + { + "name": "VisualStudio" + }, + { + "name": "Processing" + }, + { + "name": "Leiningen" + }, + { + "name": "Stella" + }, + { + "name": "Opa" + }, + { + "name": "Drupal" + }, + { + "name": "TurboGears2" + }, + { + "name": "Idris" + }, + { + "name": "Jboss" + }, + { + "name": "CodeIgniter" + }, + { + "name": "Qooxdoo" + }, + { + "name": "Waf" + }, + { + "name": "Sass" + }, + { + "name": "Lua" + }, + { + "name": "Clojure" + }, + { + "name": "IGORPro" + }, + { + "name": "Gradle" + }, + { + "name": "Archives" + }, + { + "name": "SynopsysVCS" + }, + { + "name": "Ninja" + }, + { + "name": "Tags" + }, + { + "name": "OSX" + }, + { + "name": "Dreamweaver" + }, + { + "name": "CodeKit" + }, + { + "name": "NotepadPP" + }, + { + "name": "VisualStudioCode" + }, + { + "name": "Mercurial" + }, + { + "name": "BricxCC" + }, + { + "name": "DartEditor" + }, + { + "name": "Eclipse" + }, + { + "name": "Cloud9" + }, + { + "name": "TortoiseGit" + }, + { + "name": "NetBeans" + }, + { + "name": "GPG" + }, + { + "name": "Espresso" + }, + { + "name": "Redcar" + }, + { + "name": "Xcode" + }, + { + "name": "Matlab" + }, + { + "name": "LyX" + }, + { + "name": "SlickEdit" + }, + { + "name": "Dropbox" + }, + { + "name": "CVS" + }, + { + "name": "Calabash" + }, + { + "name": "JDeveloper" + }, + { + "name": "Vagrant" + }, + { + "name": "IPythonNotebook" + }, + { + "name": "TextMate" + }, + { + "name": "Ensime" + }, + { + "name": "WebMethods" + }, + { + "name": "VirtualEnv" + }, + { + "name": "Emacs" + }, + { + "name": "Momentics" + }, + { + "name": "JetBrains" + }, + { + "name": "SublimeText" + }, + { + "name": "Kate" + }, + { + "name": "ModelSim" + }, + { + "name": "Redis" + }, + { + "name": "KDevelop4" + }, + { + "name": "Bazaar" + }, + { + "name": "Linux" + }, + { + "name": "Windows" + }, + { + "name": "XilinxISE" + }, + { + "name": "Lazarus" + }, + { + "name": "EiffelStudio" + }, + { + "name": "Anjuta" + }, + { + "name": "Vim" + }, + { + "name": "Otto" + }, + { + "name": "MicrosoftOffice" + }, + { + "name": "LibreOffice" + }, + { + "name": "SBT" + }, + { + "name": "MonoDevelop" + }, + { + "name": "SVN" + }, + { + "name": "FlexBuilder" + } +] +``` + +## Single gitignore template + +Get a single gitignore template. + +``` +GET /templates/gitignores/:key +``` + +| Attribute | Type | Required | Description | +| ---------- | ------ | -------- | ----------- | +| `key` | string | yes | The key of the gitignore template | + +```bash +curl https://gitlab.example.com/api/v3/templates/gitignores/Ruby +``` + +Example response: + +```json +{ + "name": "Ruby", + "content": "*.gem\n*.rbc\n/.config\n/coverage/\n/InstalledFiles\n/pkg/\n/spec/reports/\n/spec/examples.txt\n/test/tmp/\n/test/version_tmp/\n/tmp/\n\n# Used by dotenv library to load environment variables.\n# .env\n\n## Specific to RubyMotion:\n.dat*\n.repl_history\nbuild/\n*.bridgesupport\nbuild-iPhoneOS/\nbuild-iPhoneSimulator/\n\n## Specific to RubyMotion (use of CocoaPods):\n#\n# We recommend against adding the Pods directory to your .gitignore. However\n# you should judge for yourself, the pros and cons are mentioned at:\n# https://guides.cocoapods.org/using/using-cocoapods.html#should-i-check-the-pods-directory-into-source-control\n#\n# vendor/Pods/\n\n## Documentation cache and generated files:\n/.yardoc/\n/_yardoc/\n/doc/\n/rdoc/\n\n## Environment normalization:\n/.bundle/\n/vendor/bundle\n/lib/bundler/man/\n\n# for a library or gem, you might want to ignore these files since the code is\n# intended to run in multiple environments; otherwise, check them in:\n# Gemfile.lock\n# .ruby-version\n# .ruby-gemset\n\n# unless supporting rvm < 1.11.0 or doing something fancy, ignore this:\n.rvmrc\n" +} +``` diff --git a/doc/api/templates/gitlab_ci_ymls.md b/doc/api/templates/gitlab_ci_ymls.md new file mode 100644 index 00000000000..e120016fbe6 --- /dev/null +++ b/doc/api/templates/gitlab_ci_ymls.md @@ -0,0 +1,120 @@ +# GitLab CI YMLs + +## List GitLab CI YML templates + +Get all GitLab CI YML templates. + +``` +GET /templates/gitlab_ci_ymls +``` + +```bash +curl https://gitlab.example.com/api/v3/templates/gitlab_ci_ymls +``` + +Example response: + +```json +[ + { + "name": "C++" + }, + { + "name": "Docker" + }, + { + "name": "Elixir" + }, + { + "name": "LaTeX" + }, + { + "name": "Grails" + }, + { + "name": "Rust" + }, + { + "name": "Nodejs" + }, + { + "name": "Ruby" + }, + { + "name": "Scala" + }, + { + "name": "Maven" + }, + { + "name": "Harp" + }, + { + "name": "Pelican" + }, + { + "name": "Hyde" + }, + { + "name": "Nanoc" + }, + { + "name": "Octopress" + }, + { + "name": "JBake" + }, + { + "name": "HTML" + }, + { + "name": "Hugo" + }, + { + "name": "Metalsmith" + }, + { + "name": "Hexo" + }, + { + "name": "Lektor" + }, + { + "name": "Doxygen" + }, + { + "name": "Brunch" + }, + { + "name": "Jekyll" + }, + { + "name": "Middleman" + } +] +``` + +## Single GitLab CI YML template + +Get a single GitLab CI YML template. + +``` +GET /templates/gitlab_ci_ymls/:key +``` + +| Attribute | Type | Required | Description | +| ---------- | ------ | -------- | ----------- | +| `key` | string | yes | The key of the GitLab CI YML template | + +```bash +curl https://gitlab.example.com/api/v3/templates/gitlab_ci_ymls/Ruby +``` + +Example response: + +```json +{ + "name": "Ruby", + "content": "# This file is a template, and might need editing before it works on your project.\n# Official language image. Look for the different tagged releases at:\n# https://hub.docker.com/r/library/ruby/tags/\nimage: \"ruby:2.3\"\n\n# Pick zero or more services to be used on all builds.\n# Only needed when using a docker container to run your tests in.\n# Check out: http://docs.gitlab.com/ce/ci/docker/using_docker_images.html#what-is-service\nservices:\n - mysql:latest\n - redis:latest\n - postgres:latest\n\nvariables:\n POSTGRES_DB: database_name\n\n# Cache gems in between builds\ncache:\n paths:\n - vendor/ruby\n\n# This is a basic example for a gem or script which doesn't use\n# services such as redis or postgres\nbefore_script:\n - ruby -v # Print out ruby version for debugging\n # Uncomment next line if your rails app needs a JS runtime:\n # - apt-get update -q && apt-get install nodejs -yqq\n - gem install bundler --no-ri --no-rdoc # Bundler is not installed with the image\n - bundle install -j $(nproc) --path vendor # Install dependencies into ./vendor/ruby\n\n# Optional - Delete if not using `rubocop`\nrubocop:\n script:\n - rubocop\n\nrspec:\n script:\n - rspec spec\n\nrails:\n variables:\n DATABASE_URL: \"postgresql://postgres:postgres@postgres:5432/$POSTGRES_DB\"\n script:\n - bundle exec rake db:migrate\n - bundle exec rake db:seed\n - bundle exec rake test\n" +} +``` diff --git a/doc/api/licenses.md b/doc/api/templates/licenses.md index ed26d1fb7fb..ae7218cf1bd 100644 --- a/doc/api/licenses.md +++ b/doc/api/templates/licenses.md @@ -5,7 +5,7 @@ Get all license templates. ``` -GET /licenses +GET /templates/licenses ``` | Attribute | Type | Required | Description | @@ -13,7 +13,7 @@ GET /licenses | `popular` | boolean | no | If passed, returns only popular licenses | ```bash -curl https://gitlab.example.com/api/v3/licenses?popular=1 +curl https://gitlab.example.com/api/v3/templates/licenses?popular=1 ``` Example response: @@ -102,7 +102,7 @@ Get a single license template. You can pass parameters to replace the license placeholder. ``` -GET /licenses/:key +GET /templates/licenses/:key ``` | Attribute | Type | Required | Description | @@ -116,7 +116,7 @@ If you omit the `fullname` parameter but authenticate your request, the name of the authenticated user will be used to replace the copyright holder placeholder. ```bash -curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/licenses/mit?project=My+Cool+Project +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/templates/licenses/mit?project=My+Cool+Project ``` Example response: diff --git a/doc/api/todos.md b/doc/api/todos.md index 0cd644dfd2f..a5e81801024 100644 --- a/doc/api/todos.md +++ b/doc/api/todos.md @@ -44,7 +44,7 @@ Example Response: "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/root" + "web_url": "https://gitlab.example.com/root" }, "action_name": "marked", "target_type": "MergeRequest", @@ -67,7 +67,7 @@ Example Response: "id": 12, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/a0d477b3ea21970ce6ffcbb817b0b435?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/craig_rutherford" + "web_url": "https://gitlab.example.com/craig_rutherford" }, "assignee": { "name": "Administrator", @@ -75,7 +75,7 @@ Example Response: "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/root" + "web_url": "https://gitlab.example.com/root" }, "source_project_id": 2, "target_project_id": 2, @@ -117,7 +117,7 @@ Example Response: "id": 12, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/a0d477b3ea21970ce6ffcbb817b0b435?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/craig_rutherford" + "web_url": "https://gitlab.example.com/craig_rutherford" }, "action_name": "assigned", "target_type": "MergeRequest", @@ -140,7 +140,7 @@ Example Response: "id": 12, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/a0d477b3ea21970ce6ffcbb817b0b435?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/craig_rutherford" + "web_url": "https://gitlab.example.com/craig_rutherford" }, "assignee": { "name": "Administrator", @@ -148,7 +148,7 @@ Example Response: "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/root" + "web_url": "https://gitlab.example.com/root" }, "source_project_id": 2, "target_project_id": 2, @@ -215,7 +215,7 @@ Example Response: "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/root" + "web_url": "https://gitlab.example.com/root" }, "action_name": "marked", "target_type": "MergeRequest", @@ -238,7 +238,7 @@ Example Response: "id": 12, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/a0d477b3ea21970ce6ffcbb817b0b435?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/craig_rutherford" + "web_url": "https://gitlab.example.com/craig_rutherford" }, "assignee": { "name": "Administrator", @@ -246,7 +246,7 @@ Example Response: "id": 1, "state": "active", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "https://gitlab.example.com/u/root" + "web_url": "https://gitlab.example.com/root" }, "source_project_id": 2, "target_project_id": 2, diff --git a/doc/api/users.md b/doc/api/users.md index 2bec20d1aa6..1f923d57e98 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -20,7 +20,7 @@ GET /users "name": "John Smith", "state": "active", "avatar_url": "http://localhost:3000/uploads/user/avatar/1/cd8.jpeg", - "web_url": "http://localhost:3000/u/john_smith" + "web_url": "http://localhost:3000/john_smith" }, { "id": 2, @@ -28,7 +28,7 @@ GET /users "name": "Jack Smith", "state": "blocked", "avatar_url": "http://gravatar.com/../e32131cd8.jpeg", - "web_url": "http://localhost:3000/u/jack_smith" + "web_url": "http://localhost:3000/jack_smith" } ] ``` @@ -48,7 +48,7 @@ GET /users "name": "John Smith", "state": "active", "avatar_url": "http://localhost:3000/uploads/user/avatar/1/index.jpg", - "web_url": "http://localhost:3000/u/john_smith", + "web_url": "http://localhost:3000/john_smith", "created_at": "2012-05-23T08:00:58Z", "is_admin": false, "admin": false, @@ -84,7 +84,7 @@ GET /users "name": "Jack Smith", "state": "blocked", "avatar_url": "http://localhost:3000/uploads/user/avatar/2/index.jpg", - "web_url": "http://localhost:3000/u/jack_smith", + "web_url": "http://localhost:3000/jack_smith", "created_at": "2012-05-23T08:01:01Z", "is_admin": false, "admin": false, @@ -147,7 +147,7 @@ Parameters: "name": "John Smith", "state": "active", "avatar_url": "http://localhost:3000/uploads/user/avatar/1/cd8.jpeg", - "web_url": "http://localhost:3000/u/john_smith", + "web_url": "http://localhost:3000/john_smith", "created_at": "2012-05-23T08:00:58Z", "is_admin": false, "admin": false, @@ -181,7 +181,7 @@ Parameters: "name": "John Smith", "state": "active", "avatar_url": "http://localhost:3000/uploads/user/avatar/1/index.jpg", - "web_url": "http://localhost:3000/u/john_smith", + "web_url": "http://localhost:3000/john_smith", "created_at": "2012-05-23T08:00:58Z", "is_admin": false, "admin": false, @@ -307,7 +307,7 @@ GET /user "name": "John Smith", "state": "active", "avatar_url": "http://localhost:3000/uploads/user/avatar/1/index.jpg", - "web_url": "http://localhost:3000/u/john_smith", + "web_url": "http://localhost:3000/john_smith", "created_at": "2012-05-23T08:00:58Z", "is_admin": false, "admin": false, @@ -644,3 +644,149 @@ Parameters: Will return `200 OK` on success, `404 User Not Found` is user cannot be found or `403 Forbidden` when trying to unblock a user blocked by LDAP synchronization. + +### Get user contribution events + +Get the contribution events for the specified user, sorted from newest to oldest. + +``` +GET /users/:id/events +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | The ID of the user | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/users/:id/events +``` + +Example response: + +```json +[ + { + "title": null, + "project_id": 15, + "action_name": "closed", + "target_id": 830, + "target_type": "Issue", + "author_id": 1, + "data": null, + "target_title": "Public project search field", + "author": { + "name": "Dmitriy Zaporozhets", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", + "web_url": "http://localhost:3000/root" + }, + "author_username": "root" + }, + { + "title": null, + "project_id": 15, + "action_name": "opened", + "target_id": null, + "target_type": null, + "author_id": 1, + "author": { + "name": "Dmitriy Zaporozhets", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", + "web_url": "http://localhost:3000/root" + }, + "author_username": "john", + "data": { + "before": "50d4420237a9de7be1304607147aec22e4a14af7", + "after": "c5feabde2d8cd023215af4d2ceeb7a64839fc428", + "ref": "refs/heads/master", + "user_id": 1, + "user_name": "Dmitriy Zaporozhets", + "repository": { + "name": "gitlabhq", + "url": "git@dev.gitlab.org:gitlab/gitlabhq.git", + "description": "GitLab: self hosted Git management software. \r\nDistributed under the MIT License.", + "homepage": "https://dev.gitlab.org/gitlab/gitlabhq" + }, + "commits": [ + { + "id": "c5feabde2d8cd023215af4d2ceeb7a64839fc428", + "message": "Add simple search to projects in public area", + "timestamp": "2013-05-13T18:18:08+00:00", + "url": "https://dev.gitlab.org/gitlab/gitlabhq/commit/c5feabde2d8cd023215af4d2ceeb7a64839fc428", + "author": { + "name": "Dmitriy Zaporozhets", + "email": "dmitriy.zaporozhets@gmail.com" + } + } + ], + "total_commits_count": 1 + }, + "target_title": null + }, + { + "title": null, + "project_id": 15, + "action_name": "closed", + "target_id": 840, + "target_type": "Issue", + "author_id": 1, + "data": null, + "target_title": "Finish & merge Code search PR", + "author": { + "name": "Dmitriy Zaporozhets", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", + "web_url": "http://localhost:3000/root" + }, + "author_username": "root" + }, + { + "title": null, + "project_id": 15, + "action_name": "commented on", + "target_id": 1312, + "target_type": "Note", + "author_id": 1, + "data": null, + "target_title": null, + "created_at": "2015-12-04T10:33:58.089Z", + "note": { + "id": 1312, + "body": "What an awesome day!", + "attachment": null, + "author": { + "name": "Dmitriy Zaporozhets", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", + "web_url": "http://localhost:3000/root" + }, + "created_at": "2015-12-04T10:33:56.698Z", + "system": false, + "upvote": false, + "downvote": false, + "noteable_id": 377, + "noteable_type": "Issue" + }, + "author": { + "name": "Dmitriy Zaporozhets", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", + "web_url": "http://localhost:3000/root" + }, + "author_username": "root" + } +] +``` diff --git a/doc/api/version.md b/doc/api/version.md new file mode 100644 index 00000000000..287d17cf97f --- /dev/null +++ b/doc/api/version.md @@ -0,0 +1,23 @@ +# Version API + +>**Note:** This feature was introduced in GitLab 8.13 + +Retrieve version information for this GitLab instance. Responds `200 OK` for +authenticated users. + +``` +GET /version +``` + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/version +``` + +Example response: + +```json +{ + "version": "8.13.0-pre", + "revision": "4e963fe" +} +``` diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index 0f64137a8a9..79bbe8421c6 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -188,7 +188,7 @@ In order to do that, follow the steps: image = "docker:latest" privileged = false disable_cache = false - volumes = ["/var/run/docker.sock", "/cache"] + volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"] [runners.cache] Insecure = false ``` diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index a849905ac6b..aba77490915 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -37,7 +37,7 @@ The registered runner will use the `ruby:2.1` docker image and will run two services, `postgres:latest` and `mysql:latest`, both of which will be accessible during the build process. -## What is image +## What is an image The `image` keyword is the name of the docker image that is present in the local Docker Engine (list all images with `docker images`) or any image that @@ -47,7 +47,7 @@ Hub please read the [Docker Fundamentals][] documentation. In short, with `image` we refer to the docker image, which will be used to create a container on which your build will run. -## What is service +## What is a service The `services` keyword defines just another docker image that is run during your build and is linked to the docker image that the `image` keyword defines. @@ -61,7 +61,7 @@ time the project is built. You can see some widely used services examples in the relevant documentation of [CI services examples](../services/README.md). -### How is service linked to the build +### How services are linked to the build To better understand how the container linking works, read [Linking containers together][linking-containers]. @@ -221,7 +221,7 @@ time. *Note: The following commands are run without root privileges. You should be able to run docker with your regular user account.* -First start with creating a file named `build script`: +First start with creating a file named `build_script`: ```bash cat <<EOF > build_script diff --git a/doc/ci/environments.md b/doc/ci/environments.md index d85b8a34ced..e070302fb82 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -14,6 +14,19 @@ Defining environments in a project's `.gitlab-ci.yml` lets developers track Deployments are created when [jobs] deploy versions of code to [environments]. +### Checkout deployments locally + +Since 8.13, a reference in the git repository is saved for each deployment. So +knowing what the state is of your current environments is only a `git fetch` +away. + +In your git config, append the `[remote "<your-remote>"]` block with an extra +fetch line: + +``` +fetch = +refs/environments/*:refs/remotes/origin/environments/* +``` + ## Defining environments You can create and delete environments manually in the web interface, but we diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index 40f0165deef..ffc310ec8c7 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -11,7 +11,9 @@ Apart from those, here is an collection of tutorials and guides on setting up yo - [Test and deploy a Python application to Heroku](test-and-deploy-python-application-to-heroku.md) - [Test a Clojure application](test-clojure-application.md) - [Test a Scala application](test-scala-application.md) +- [Test a Phoenix application](test-phoenix-application.md) - [Using `dpl` as deployment tool](deployment/README.md) +- [Example project that shows how to use Review Apps](https://gitlab.com/gitlab-examples/review-apps-nginx/) - [Blog post about using GitLab CI for iOS projects](https://about.gitlab.com/2016/03/10/setting-up-gitlab-ci-for-ios-projects/) - [Repositories with examples for various languages](https://gitlab.com/groups/gitlab-examples) - [The .gitlab-ci.yml file for GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml) diff --git a/doc/ci/examples/test-phoenix-application.md b/doc/ci/examples/test-phoenix-application.md new file mode 100644 index 00000000000..150698ca04b --- /dev/null +++ b/doc/ci/examples/test-phoenix-application.md @@ -0,0 +1,56 @@ +## Test a Phoenix application + +This example demonstrates the integration of Gitlab CI with Phoenix, Elixir and +Postgres. + +### Add `.gitlab-ci.yml` file to project + +The following `.gitlab-ci.yml` should be added in the root of your +repository to trigger CI: + +```yaml +image: elixir:1.3 + +services: + - postgres:9.6 + +variables: + MIX_ENV: "test" + +before_script: + # Setup phoenix dependencies + - apt-get update + - apt-get install -y postgresql-client + - mix local.hex --force + - mix deps.get --only test + - mix ecto.reset + +test: + script: + - mix test +``` + +The variables will set the Mix environment to "test". The +`before_script` will install `psql`, some Phoenix dependencies, and will also +run your migrations. + +Finally, the test `script` will run your tests. + +### Update the Config Settings + +In `config/test.exs`, update the database hostname: + +```elixir +config :my_app, MyApp.Repo, + hostname: if(System.get_env("CI"), do: "postgres", else: "localhost"), +``` + +### Add the Migrations Folder + +If you do not have any migrations yet, you will need to create an empty +`.gitkeep` file in `priv/repo/migrations`. + +### Sources + +- https://medium.com/@nahtnam/using-phoenix-on-gitlab-ci-5a51eec81142 +- https://davejlong.com/ci-with-phoenix-and-gitlab/ diff --git a/doc/ci/pipelines.md b/doc/ci/pipelines.md index ca9b986a060..729c1dc8c0d 100644 --- a/doc/ci/pipelines.md +++ b/doc/ci/pipelines.md @@ -31,6 +31,8 @@ project. ## Seeing build status Clicking on a pipeline will show the builds that were run for that pipeline. +Clicking on an individual build will show you its build trace, and allow you to +cancel the build, retry it, or erase the build trace. ## Badges diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index b78422f6d0e..84048f1d25f 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -4,7 +4,7 @@ > **Note**: GitLab 8.12 has a completely redesigned build permissions system. -Read all about the [new model and its implications][../../user/project/new_ci_build_permissions_model.md#build-triggers]. +Read all about the [new model and its implications](../../user/project/new_ci_build_permissions_model.md#build-triggers). Triggers can be used to force a rebuild of a specific branch, tag or commit, with an API call. diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 22d67bd9964..a4c3a731a20 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -48,6 +48,7 @@ The `API_TOKEN` will take the Secure Variable value: `SECURE`. | **CI_RUNNER_ID** | 8.10 | 0.5 | The unique id of runner being used | | **CI_RUNNER_DESCRIPTION** | 8.10 | 0.5 | The description of the runner as saved in GitLab | | **CI_RUNNER_TAGS** | 8.10 | 0.5 | The defined runner tags | +| **CI_DEBUG_TRACE** | all | 1.7 | Whether [debug tracing](#debug-tracing) is enabled | | **GITLAB_USER_ID** | 8.12 | all | The id of the user who started the build | | **GITLAB_USER_EMAIL** | 8.12 | all | The email of the user who started the build | @@ -105,6 +106,39 @@ Variables can be defined at a global level, but also at a job level. More information about Docker integration can be found in [Using Docker Images](../docker/using_docker_images.md). +#### Debug tracing + +> **WARNING:** Enabling debug tracing can have severe security implications. The + output **will** contain the content of all your secure variables and any other + secrets! The output **will** be uploaded to the GitLab server and made visible + in build traces! + +By default, GitLab Runner hides most of the details of what it is doing when +processing a job. This behaviour keeps build traces short, and prevents secrets +from being leaked into the trace unless your script writes them to the screen. + +If a job isn't working as expected, this can make the problem difficult to +investigate; in these cases, you can enable debug tracing in `.gitlab-ci.yml`. +Available on GitLab Runner v1.7+, this feature enables the shell's execution +trace, resulting in a verbose build trace listing all commands that were run, +variables that were set, etc. + +Before enabling this, you should ensure builds are visible to +[team members only](../../../user/permissions.md#project-features). You should +also [erase](../pipelines.md#seeing-build-traces) all generated build traces +before making them visible again. + +To enable debug traces, set the `CI_DEBUG_TRACE` variable to `true`: + +```yaml +job1: + variables: + CI_DEBUG_TRACE: "true" +``` + +The [example project](https://gitlab.com/gitlab-examples/ci-debug-trace) +demonstrates a working configuration, including build trace examples. + ### User-defined variables (Secure Variables) **This feature requires GitLab Runner 0.4.0 or higher** diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 16868554c1f..5c0e1c44e3f 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -146,20 +146,25 @@ variables: ``` These variables can be later used in all executed commands and scripts. - The YAML-defined variables are also set to all created service containers, -thus allowing to fine tune them. +thus allowing to fine tune them. Variables can be also defined on a +[job level](#job-variables). -Variables can be also defined on [job level](#job-variables). +Except for the user defined variables, there are also the ones set up by the +Runner itself. One example would be `CI_BUILD_REF_NAME` which has the value of +the branch or tag name for which project is built. Apart from the variables +you can set in `.gitlab-ci.yml`, there are also the so called secret variables +which can be set in GitLab's UI. -[Learn more about variables.](../variables/README.md) +[Learn more about variables.][variables] ### cache > Introduced in GitLab Runner v0.7.0. `cache` is used to specify a list of files and directories which should be -cached between builds. +cached between builds. You can only use paths that are within the project +workspace. **By default the caching is enabled per-job and per-branch.** @@ -540,20 +545,29 @@ An example usage of manual actions is deployment to production. > Introduced in GitLab 8.9. -`environment` is used to define that a job deploys to a specific [environment]. -This allows easy tracking of all deployments to your environments straight from -GitLab. +> You can read more about environments and find more examples in the +[documentation about environments][environment]. +`environment` is used to define that a job deploys to a specific environment. If `environment` is specified and no environment under that name exists, a new one will be created automatically. -The `environment` name must contain only letters, digits, '-', '_', '/', '$', '{', '}' and spaces. Common -names are `qa`, `staging`, and `production`, but you can use whatever name works -with your workflow. +The `environment` name can contain: ---- +- letters +- digits +- spaces +- `-` +- `_` +- `/` +- `$` +- `{` +- `}` -**Example configurations** +Common names are `qa`, `staging`, and `production`, but you can use whatever +name works with your workflow. + +In its simplest form, the `environment` keyword can be defined like: ``` deploy to production: @@ -562,37 +576,134 @@ deploy to production: environment: production ``` -The `deploy to production` job will be marked as doing deployment to -`production` environment. +In the above example, the `deploy to production` job will be marked as doing a +deployment to the `production` environment. + +#### environment:name + +> Introduced in GitLab 8.11. + +>**Note:** +Before GitLab 8.11, the name of an environment could be defined as a string like +`environment: production`. The recommended way now is to define it under the +`name` keyword. + +Instead of defining the name of the environment right after the `environment` +keyword, it is also possible to define it as a separate value. For that, use +the `name` keyword under `environment`: + +``` +deploy to production: + stage: deploy + script: git push production HEAD:master + environment: + name: production +``` + +#### environment:url + +> Introduced in GitLab 8.11. + +>**Note:** +Before GitLab 8.11, the URL could be added only in GitLab's UI. The +recommended way now is to define it in `.gitlab-ci.yml`. + +This is an optional value that when set, it exposes buttons in various places +in GitLab which when clicked take you to the defined URL. + +In the example below, if the job finishes successfully, it will create buttons +in the merge requests and in the environments/deployments pages which will point +to `https://prod.example.com`. + +``` +deploy to production: + stage: deploy + script: git push production HEAD:master + environment: + name: production + url: https://prod.example.com +``` + +#### environment:on_stop + +> [Introduced][ce-6669] in GitLab 8.13. + +Closing (stoping) environments can be achieved with the `on_stop` keyword defined under +`environment`. It declares a different job that runs in order to close +the environment. + +Read the `environment:action` section for an example. + +#### environment:action + +> [Introduced][ce-6669] in GitLab 8.13. + +The `action` keyword is to be used in conjunction with `on_stop` and is defined +in the job that is called to close the environment. + +Take for instance: + +```yaml +review_app: + stage: deploy + script: make deploy-app + environment: + name: review + on_stop: stop_review_app + +stop_review_app: + stage: deploy + script: make delete-app + when: manual + environment: + name: review + action: stop +``` + +In the above example we set up the `review_app` job to deploy to the `review` +environment, and we also defined a new `stop_review_app` job under `on_stop`. +Once the `review_app` job is successfully finished, it will trigger the +`stop_review_app` job based on what is defined under `when`. In this case we +set it up to `manual` so it will need a [manual action](#manual-actions) via +GitLab's web interface in order to run. + +The `stop_review_app` job is **required** to have the following keywords defined: + +- `when` - [reference](#when) +- `environment:name` +- `environment:action` #### dynamic environments > [Introduced][ce-6323] in GitLab 8.12 and GitLab Runner 1.6. `environment` can also represent a configuration hash with `name` and `url`. -These parameters can use any of the defined CI [variables](#variables) +These parameters can use any of the defined [CI variables](#variables) (including predefined, secure variables and `.gitlab-ci.yml` variables). -The common use case is to create dynamic environments for branches and use them -as review apps. - ---- - -**Example configurations** +For example: ``` deploy as review app: stage: deploy - script: ... + script: make deploy environment: name: review-apps/$CI_BUILD_REF_NAME url: https://$CI_BUILD_REF_NAME.review.example.com/ ``` The `deploy as review app` job will be marked as deployment to dynamically -create the `review-apps/branch-name` environment. +create the `review-apps/$CI_BUILD_REF_NAME` environment, which `$CI_BUILD_REF_NAME` +is an [environment variable][variables] set by the Runner. If for example the +`deploy as review app` job was run in a branch named `pow`, this environment +should be accessible under `https://pow.review.example.com/`. -This environment should be accessible under `https://branch-name.review.example.com/`. +This of course implies that the underlying server which hosts the application +is properly configured. + +The common use case is to create dynamic environments for branches and use them +as Review Apps. You can see a simple example using Review Apps at +https://gitlab.com/gitlab-examples/review-apps-nginx/. ### artifacts @@ -604,8 +715,8 @@ This environment should be accessible under `https://branch-name.review.example. > - Build artifacts are only collected for successful builds by default. `artifacts` is used to specify a list of files and directories which should be -attached to the build after success. To pass artifacts between different builds, -see [dependencies](#dependencies). +attached to the build after success. You can only use paths that are within the +project workspace. To pass artifacts between different builds, see [dependencies](#dependencies). Below are some examples. @@ -858,27 +969,45 @@ job: ## Git Strategy -> Introduced in GitLab 8.9 as an experimental feature. May change in future - releases or be removed completely. +> Introduced in GitLab 8.9 as an experimental feature. May change or be removed + completely in future releases. `GIT_STRATEGY=none` requires GitLab Runner + v1.7+. + +You can set the `GIT_STRATEGY` used for getting recent application code, either +in the global [`variables`](#variables) section or the [`variables`](#job-variables) +section for individual jobs. If left unspecified, the default from project +settings will be used. -You can set the `GIT_STRATEGY` used for getting recent application code. `clone` -is slower, but makes sure you have a clean directory before every build. `fetch` -is faster. `GIT_STRATEGY` can be specified in the global `variables` section or -in the `variables` section for individual jobs. If it's not specified, then the -default from project settings will be used. +There are three possible values: `clone`, `fetch`, and `none`. + +`clone` is the slowest option. It clones the repository from scratch for every +job, ensuring that the project workspace is always pristine. ``` variables: GIT_STRATEGY: clone ``` -or +`fetch` is faster as it re-uses the project workspace (falling back to `clone` +if it doesn't exist). `git clean` is used to undo any changes made by the last +job, and `git fetch` is used to retrieve commits made since the last job ran. ``` variables: GIT_STRATEGY: fetch ``` +`none` also re-uses the project workspace, but skips all Git operations +(including GitLab Runner's pre-clone script, if present). It is mostly useful +for jobs that operate exclusively on artifacts (e.g., `deploy`). Git repository +data may be present, but it is certain to be out of date, so you should only +rely on files brought into the project workspace from cache or artifacts. + +``` +variables: + GIT_STRATEGY: none +``` + ## Shallow cloning > Introduced in GitLab 8.9 as an experimental feature. May change in future @@ -1084,3 +1213,5 @@ CI with various languages. [examples]: ../examples/README.md [ce-6323]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6323 [environment]: ../environments.md +[ce-6669]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6669 +[variables]: ../variables/README.md diff --git a/doc/container_registry/README.md b/doc/container_registry/README.md index d7740647a91..fe3e4681ba7 100644 --- a/doc/container_registry/README.md +++ b/doc/container_registry/README.md @@ -1,98 +1 @@ -# GitLab Container Registry - -> [Introduced][ce-4040] in GitLab 8.8. Docker Registry manifest -`v1` support was added in GitLab 8.9 to support Docker versions earlier than 1.10. - -> **Note:** -This document is about the user guide. To learn how to enable GitLab Container -Registry across your GitLab instance, visit the -[administrator documentation](../administration/container_registry.md). - -With the Docker Container Registry integrated into GitLab, every project can -have its own space to store its Docker images. - -You can read more about Docker Registry at https://docs.docker.com/registry/introduction/. - ---- - -## Enable the Container Registry for your project - -1. First, ask your system administrator to enable GitLab Container Registry - following the [administration documentation](../administration/container_registry.md). - If you are using GitLab.com, this is enabled by default so you can start using - the Registry immediately. - -1. Go to your project's settings and enable the **Container Registry** feature - on your project. For new projects this might be enabled by default. For - existing projects you will have to explicitly enable it. - -  - -## Build and push images - -After you save your project's settings, you should see a new link in the -sidebar called **Container Registry**. Following this link will get you to -your project's Registry panel where you can see how to login to the Container -Registry using your GitLab credentials. - -For example if the Registry's URL is `registry.example.com`, the you should be -able to login with: - -``` -docker login registry.example.com -``` - -Building and publishing images should be a straightforward process. Just make -sure that you are using the Registry URL with the namespace and project name -that is hosted on GitLab: - -``` -docker build -t registry.example.com/group/project . -docker push registry.example.com/group/project -``` - -## Use images from GitLab Container Registry - -To download and run a container from images hosted in GitLab Container Registry, -use `docker run`: - -``` -docker run [options] registry.example.com/group/project [arguments] -``` - -For more information on running Docker containers, visit the -[Docker documentation][docker-docs]. - -## Control Container Registry from within GitLab - -GitLab offers a simple Container Registry management panel. Go to your project -and click **Container Registry** in the left sidebar. - -This view will show you all tags in your project and will easily allow you to -delete them. - - - -## Build and push images using GitLab CI - -> **Note:** -This feature requires GitLab 8.8 and GitLab Runner 1.2. - -Make sure that your GitLab Runner is configured to allow building Docker images by -following the [Using Docker Build](../ci/docker/using_docker_build.md) -and [Using the GitLab Container Registry documentation](../ci/docker/using_docker_build.md#using-the-gitlab-container-registry). - -## Limitations - -In order to use a container image from your private project as an `image:` in -your `.gitlab-ci.yml`, you have to follow the -[Using a private Docker Registry][private-docker] -documentation. This workflow will be simplified in the future. - -## Troubleshooting - -See [the GitLab Docker registry troubleshooting guide](troubleshooting.md). - -[ce-4040]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4040 -[docker-docs]: https://docs.docker.com/engine/userguide/intro/ -[private-docker]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/configuration/advanced-configuration.md#using-a-private-docker-registry +This document was moved in [user/project/container_registry](../user/project/container_registry.md). diff --git a/doc/container_registry/img/container_registry.png b/doc/container_registry/img/container_registry.png Binary files differdeleted file mode 100644 index 57d6f9f22c5..00000000000 --- a/doc/container_registry/img/container_registry.png +++ /dev/null diff --git a/doc/container_registry/img/project_feature.png b/doc/container_registry/img/project_feature.png Binary files differdeleted file mode 100644 index a59b4f82b56..00000000000 --- a/doc/container_registry/img/project_feature.png +++ /dev/null diff --git a/doc/container_registry/troubleshooting.md b/doc/container_registry/troubleshooting.md index 14c4a7d9a63..2f8cd37b488 100644 --- a/doc/container_registry/troubleshooting.md +++ b/doc/container_registry/troubleshooting.md @@ -1,141 +1 @@ -# Troubleshooting the GitLab Container Registry - -## Basic Troubleshooting - -1. Check to make sure that the system clock on your Docker client and GitLab server have - been synchronized (e.g. via NTP). - -2. If you are using an S3-backed Registry, double check that the IAM - permissions and the S3 credentials (including region) are correct. See [the - sample IAM policy](https://docs.docker.com/registry/storage-drivers/s3/) - for more details. - -3. Check the Registry logs (e.g. `/var/log/gitlab/registry/current`) and the GitLab production logs - for errors (e.g. `/var/log/gitlab/gitlab-rails/production.log`). You may be able to find clues - there. - -## Advanced Troubleshooting - ->**NOTE:** The following section is only recommended for experts. - -Sometimes it's not obvious what is wrong, and you may need to dive deeper into -the communication between the Docker client and the Registry to find out -what's wrong. We will use a concrete example in the past to illustrate how to -diagnose a problem with the S3 setup. - -### Unexpected 403 error during push - -A user attempted to enable an S3-backed Registry. The `docker login` step went -fine. However, when pushing an image, the output showed: - -``` -The push refers to a repository [s3-testing.myregistry.com:4567/root/docker-test] -dc5e59c14160: Pushing [==================================================>] 14.85 kB -03c20c1a019a: Pushing [==================================================>] 2.048 kB -a08f14ef632e: Pushing [==================================================>] 2.048 kB -228950524c88: Pushing 2.048 kB -6a8ecde4cc03: Pushing [==> ] 9.901 MB/205.7 MB -5f70bf18a086: Pushing 1.024 kB -737f40e80b7f: Waiting -82b57dbc5385: Waiting -19429b698a22: Waiting -9436069b92a3: Waiting -error parsing HTTP 403 response body: unexpected end of JSON input: "" -``` - -This error is ambiguous, as it's not clear whether the 403 is coming from the -GitLab Rails application, the Docker Registry, or something else. In this -case, since we know that since the login succeeded, we probably need to look -at the communication between the client and the Registry. - -The REST API between the Docker client and Registry is [described -here](https://docs.docker.com/registry/spec/api/). Normally, one would just -use Wireshark or tcpdump to capture the traffic and see where things went -wrong. However, since all communication between Docker clients and servers -are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even -if you know the private key. What can we do instead? - -One way would be to disable HTTPS by setting up an [insecure -Registry](https://docs.docker.com/registry/insecure/). This could introduce a -security hole and is only recommended for local testing. If you have a -production system and can't or don't want to do this, there is another way: -use mitmproxy, which stands for Man-in-the-Middle Proxy. - -### mitmproxy - -[mitmproxy](https://mitmproxy.org/) allows you to place a proxy between your -client and server to inspect all traffic. One wrinkle is that your system -needs to trust the mitmproxy SSL certificates for this to work. - -The following installation instructions assume you are running Ubuntu: - -1. Install mitmproxy (see http://docs.mitmproxy.org/en/stable/install.html) -1. Run `mitmproxy --port 9000` to generate its certificates. - Enter <kbd>CTRL</kbd>-<kbd>C</kbd> to quit. -1. Install the certificate from `~/.mitmproxy` to your system: - - ```sh - sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt - sudo update-ca-certificates - ``` - -If successful, the output should indicate that a certificate was added: - -```sh -Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done. -Running hooks in /etc/ca-certificates/update.d....done. -``` - -To verify that the certificates are properly installed, run: - -```sh -mitmproxy --port 9000 -``` - -This will run mitmproxy on port `9000`. In another window, run: - -```sh -curl --proxy http://localhost:9000 https://httpbin.org/status/200 -``` - -If everything is setup correctly, you will see information on the mitmproxy window and -no errors from the curl commands. - -### Running the Docker daemon with a proxy - -For Docker to connect through a proxy, you must start the Docker daemon with the -proper environment variables. The easiest way is to shutdown Docker (e.g. `sudo initctl stop docker`) -and then run Docker by hand. As root, run: - -```sh -export HTTP_PROXY="http://localhost:9000" -export HTTPS_PROXY="https://localhost:9000" -docker daemon --debug -``` - -This will launch the Docker daemon and proxy all connections through mitmproxy. - -### Running the Docker client - -Now that we have mitmproxy and Docker running, we can attempt to login and push -a container image. You may need to run as root to do this. For example: - -```sh -docker login s3-testing.myregistry.com:4567 -docker push s3-testing.myregistry.com:4567/root/docker-test -``` - -In the example above, we see the following trace on the mitmproxy window: - - - -The above image shows: - -* The initial PUT requests went through fine with a 201 status code. -* The 201 redirected the client to the S3 bucket. -* The HEAD request to the AWS bucket reported a 403 Unauthorized. - -What does this mean? This strongly suggests that the S3 user does not have the right -[permissions to perform a HEAD request](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html). -The solution: check the [IAM permissions again](https://docs.docker.com/registry/storage-drivers/s3/). -Once the right permissions were set, the error will go away. +This document was moved to [user/project/container_registry](../user/project/container_registry.md). diff --git a/doc/development/README.md b/doc/development/README.md index 58c00f618fa..fb6a8a5b095 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -13,7 +13,9 @@ - [SQL Migration Style Guide](migration_style_guide.md) for creating safe SQL migrations - [Testing standards and style guidelines](testing.md) - [UI guide](ui_guide.md) for building GitLab with existing CSS styles and elements -- [SQL guidelines](sql.md) for SQL guidelines +- [Frontend guidelines](frontend.md) +- [SQL guidelines](sql.md) for working with SQL queries +- [Sidekiq guidelines](sidekiq_style_guide.md) for working with Sidekiq workers ## Process diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 40ae55ab905..c5c23b5c0b8 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -34,6 +34,10 @@ request is up to one of our merge request "endbosses", denoted on the ## Having your code reviewed +Please keep in mind that code review is a process that can take multiple +iterations, and reviewers may spot things later that they may not have seen the +first time. + - The first reviewer of your code is _you_. Before you perform that first push of your shiny new branch, read through the entire diff. Does it make sense? Did you include something unrelated to the overall purpose of the changes? Did @@ -55,6 +59,7 @@ request is up to one of our merge request "endbosses", denoted on the Understand why the change is necessary (fixes a bug, improves the user experience, refactors the existing code). Then: +- Try to be thorough in your reviews to reduce the number of iterations. - Communicate which ideas you feel strongly about and those you don't. - Identify ways to simplify the code while still solving the problem. - Offer alternative implementations, but assume the author already considered @@ -64,8 +69,10 @@ experience, refactors the existing code). Then: someone else would be confused by it as well. - After a round of line notes, it can be helpful to post a summary note such as "LGTM :thumbsup:", or "Just a couple things to address." -- Avoid accepting a merge request before the build succeeds ("Merge when build - succeeds" is fine). +- Avoid accepting a merge request before the build succeeds. Of course, "Merge + When Build Succeeds" (MWBS) is fine. +- If you set the MR to "Merge When Build Succeeds", you should take over + subsequent revisions for anything that would be spotted after that. ## Credits diff --git a/doc/development/doc_styleguide.md b/doc/development/doc_styleguide.md index 39b801f761d..f07d2c9af2d 100644 --- a/doc/development/doc_styleguide.md +++ b/doc/development/doc_styleguide.md @@ -95,10 +95,10 @@ merge request. someone in the Merge Request - When introducing a new document, be careful for the headings to be grammatically and syntactically correct. It is advised to mention one or all - of the following GitLab members for a review: `@axil`, `@rspeicher`, - `@dblessing`, `@ashleys`. This is to ensure that no document - with wrong heading is going live without an audit, thus preventing dead links - and redirection issues when corrected + of the following GitLab members for a review: `@axil`, `@rspeicher`, `@marcia`, + `@SeanPackham`. This is to ensure that no document with wrong heading is going + live without an audit, thus preventing dead links and redirection issues when + corrected - Leave exactly one newline after a heading ## Links @@ -314,6 +314,29 @@ In this case: - different highlighting languages are used for each config in the code block - the [references](#references) guide is used for reconfigure/restart +## Fake tokens + +There may be times where a token is needed to demonstrate an API call using +cURL or a secret variable used in CI. It is strongly advised not to use real +tokens in documentation even if the probability of a token being exploited is +low. + +You can use the following fake tokens as examples. + +| **Token type** | **Token value** | +| --------------------- | --------------------------------- | +| Private user token | `9koXpg98eAheJpvBs5tK` | +| Personal access token | `n671WNGecHugsdEDPsyo` | +| Application ID | `2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6` | +| Application secret | `04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df` | +| Secret CI variable | `Li8j-mLUVA3eZYjPfd_H` | +| Specific Runner token | `yrnZW46BrtBFqM7xDzE7dddd` | +| Shared Runner token | `6Vk7ZsosqQyfreAxXTZr` | +| Trigger token | `be20d8dcc028677c931e04f3871a9b` | +| Webhook secret token | `6XhDroRcYPM5by_h-HLY` | +| Health check token | `Tu7BgjR9qeZTEyRzGG2P` | +| Request profile token | `7VgpS4Ax5utVD2esNstz` | + ## API Here is a list of must-have items. Use them in the exact order that appears @@ -449,4 +472,4 @@ curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --data "domain [doc-restart]: ../administration/restart_gitlab.md "GitLab restart documentation" [ce-3349]: https://gitlab.com/gitlab-org/gitlab-ce/issues/3349 "Documentation restructure" [graffle]: https://gitlab.com/gitlab-org/gitlab-design/blob/d8d39f4a87b90fb9ae89ca12dc565347b4900d5e/production/resources/gitlab-map.graffle -[gitlab-map]: https://gitlab.com/gitlab-org/gitlab-design/raw/master/production/resources/gitlab-map.png +[gitlab-map]: https://gitlab.com/gitlab-org/gitlab-design/raw/master/production/resources/gitlab-map.png
\ No newline at end of file diff --git a/doc/development/frontend.md b/doc/development/frontend.md new file mode 100644 index 00000000000..56c8516508e --- /dev/null +++ b/doc/development/frontend.md @@ -0,0 +1,236 @@ +# Frontend Development Guidelines + +This document describes various guidelines to ensure consistency and quality +across GitLab's frontend team. + +## Overview + +GitLab is built on top of [Ruby on Rails][rails] using [Haml][haml] with +[Hamlit][hamlit]. Be wary of [the limitations that come with using +Hamlit][hamlit-limits]. We also use [SCSS][scss] and plain JavaScript with +[ES6 by way of Babel][es6]. + +The asset pipeline is [Sprockets][sprockets], which handles the concatenation, +minification, and compression of our assets. + +[jQuery][jquery] is used throughout the application's JavaScript, with +[Vue.js][vue] for particularly advanced, dynamic elements. + +### Vue + +For more complex frontend features, we recommend using Vue.js. It shares +some ideas with React.js as well as Angular. + +To get started with Vue, read through [their documentation][vue-docs]. + +## Performance + +### Resources + +- [WebPage Test][web-page-test] for testing site loading time and size. +- [Google PageSpeed Insights][pagespeed-insights] grades web pages and provides feedback to improve the page. +- [Profiling with Chrome DevTools][google-devtools-profiling] +- [Browser Diet][browser-diet] is a community-built guide that catalogues practical tips for improving web page performance. + +### Page-specific JavaScript + +Certain pages may require the use of a third party library, such as [d3][d3] for +the User Activity Calendar and [Chart.js][chartjs] for the Graphs pages. These +libraries increase the page size significantly, and impact load times due to +bandwidth bottlenecks and the browser needing to parse more JavaScript. + +In cases where libraries are only used on a few specific pages, we use +"page-specific JavaScript" to prevent the main `application.js` file from +becoming unnecessarily large. + +Steps to split page-specific JavaScript from the main `application.js`: + +1. Create a directory for the specific page(s), e.g. `graphs/`. +1. In that directory, create a `namespace_bundle.js` file, e.g. `graphs_bundle.js`. +1. In `graphs_bundle.js` add the line `//= require_tree .`, this adds all other files in the directory to the bundle. +1. Add any necessary libraries to `app/assets/javascripts/lib/`, all files directly descendant from this directory will be precompiled as separate assets, in this case `chart.js` would be added. +1. Add the new "bundle" file to the list of precompiled assets in +`config/application.rb`. + - For example: `config.assets.precompile << "graphs/graphs_bundle.js"`. +1. Move code reliant on these libraries into the `graphs` directory. +1. In the relevant views, add the scripts to the page with the following: + +```haml +- content_for :page_specific_javascripts do + = page_specific_javascript_tag('lib/chart.js') + = page_specific_javascript_tag('graphs/graphs_bundle.js') +``` + +The above loads `chart.js` and `graphs_bundle.js` for this page only. `chart.js` +is separated from the bundle file so it can be cached separately from the bundle +and reused for other pages that also rely on the library. For an example, see +[this Haml file][page-specific-js-example]. + +### Minimizing page size + +A smaller page size means the page loads faster (especially important on mobile +and poor connections), the page is parsed more quickly by the browser, and less +data is used for users with capped data plans. + +General tips: + +- Don't add new fonts. +- Prefer font formats with better compression, e.g. WOFF2 is better than WOFF, which is better than TTF. +- Compress and minify assets wherever possible (For CSS/JS, Sprockets does this for us). +- If some functionality can reasonably be achieved without adding extra libraries, avoid them. +- Use page-specific JavaScript as described above to dynamically load libraries that are only needed on certain pages. + +## Accessibility + +### Resources + +[Chrome Accessibility Developer Tools][chrome-accessibility-developer-tools] +are useful for testing for potential accessibility problems in GitLab. + +Accessibility best-practices and more in-depth information is available on +[the Audit Rules page][audit-rules] for the Chrome Accessibility Developer Tools. + +## Security + +### Resources + +[Mozilla’s HTTP Observatory CLI][observatory-cli] and the +[Qualys SSL Labs Server Test][qualys-ssl] are good resources for finding +potential problems and ensuring compliance with security best practices. + +<!-- Uncomment these sections when CSP/SRI are implemented. +### Content Security Policy (CSP) + +Content Security Policy is a web standard that intends to mitigate certain +forms of Cross-Site Scripting (XSS) as well as data injection. + +Content Security Policy rules should be taken into consideration when +implementing new features, especially those that may rely on connection with +external services. + +GitLab's CSP is used for the following: + +- Blocking plugins like Flash and Silverlight from running at all on our pages. +- Blocking the use of scripts and stylesheets downloaded from external sources. +- Upgrading `http` requests to `https` when possible. +- Preventing `iframe` elements from loading in most contexts. + +Some exceptions include: + +- Scripts from Google Analytics and Piwik if either is enabled. +- Connecting with GitHub, Bitbucket, GitLab.com, etc. to allow project importing. +- Connecting with Google, Twitter, GitHub, etc. to allow OAuth authentication. + +We use [the Secure Headers gem][secure_headers] to enable Content +Security Policy headers in the GitLab Rails app. + +Some resources on implementing Content Security Policy: + +- [MDN Article on CSP][mdn-csp] +- [GitHub’s CSP Journey on the GitHub Engineering Blog][github-eng-csp] +- The Dropbox Engineering Blog's series on CSP: [1][dropbox-csp-1], [2][dropbox-csp-2], [3][dropbox-csp-3], [4][dropbox-csp-4] + +### Subresource Integrity (SRI) + +Subresource Integrity prevents malicious assets from being provided by a CDN by +guaranteeing that the asset downloaded is identical to the asset the server +is expecting. + +The Rails app generates a unique hash of the asset, which is used as the +asset's `integrity` attribute. The browser generates the hash of the asset +on-load and will reject the asset if the hashes do not match. + +All CSS and JavaScript assets should use Subresource Integrity. For implementation details, +see the documentation for [the Sprockets implementation of SRI][sprockets-sri]. + +Some resources on implementing Subresource Integrity: + +- [MDN Article on SRI][mdn-sri] +- [Subresource Integrity on the GitHub Engineering Blog][github-eng-sri] + +--> + +### Including external resources + +External fonts, CSS, and JavaScript should never be used with the exception of +Google Analytics and Piwik - and only when the instance has enabled it. Assets +should always be hosted and served locally from the GitLab instance. Embedded +resources via `iframes` should never be used except in certain circumstances +such as with ReCaptcha, which cannot be used without an `iframe`. + +### Avoiding inline scripts and styles + +In order to protect users from [XSS vulnerabilities][xss], we will disable inline scripts in the future using Content Security Policy. + +While inline scripts can be useful, they're also a security concern. If +user-supplied content is unintentionally left un-sanitized, malicious users can +inject scripts into the web app. + +Inline styles should be avoided in almost all cases, they should only be used +when no alternatives can be found. This allows reusability of styles as well as +readability. + +## Style guides and linting + +See the relevant style guides for our guidelines and for information on linting: + +- [SCSS][scss-style-guide] + +## Testing + +Feature tests need to be written for all new features. Regression tests +also need to be written for all bug fixes to prevent them from occurring +again in the future. + +See [the Testing Standards and Style Guidelines](testing.md) for more +information. + +## Supported browsers + +For our currently-supported browsers, see our [requirements][requirements]. + +[rails]: http://rubyonrails.org/ +[haml]: http://haml.info/ +[hamlit]: https://github.com/k0kubun/hamlit +[hamlit-limits]: https://github.com/k0kubun/hamlit/blob/master/REFERENCE.md#limitations +[scss]: http://sass-lang.com/ +[es6]: https://babeljs.io/ +[sprockets]: https://github.com/rails/sprockets +[jquery]: https://jquery.com/ +[vue]: http://vuejs.org/ +[vue-docs]: http://vuejs.org/guide/index.html +[web-page-test]: http://www.webpagetest.org/ +[pagespeed-insights]: https://developers.google.com/speed/pagespeed/insights/ +[google-devtools-profiling]: https://developers.google.com/web/tools/chrome-devtools/profile/?hl=en +[browser-diet]: https://browserdiet.com/ +[d3]: https://d3js.org/ +[chartjs]: http://www.chartjs.org/ +[page-specific-js-example]: https://gitlab.com/gitlab-org/gitlab-ce/blob/13bb9ed77f405c5f6ee4fdbc964ecf635c9a223f/app/views/projects/graphs/_head.html.haml#L6-8 +[chrome-accessibility-developer-tools]: https://github.com/GoogleChrome/accessibility-developer-tools +[audit-rules]: https://github.com/GoogleChrome/accessibility-developer-tools/wiki/Audit-Rules +[observatory-cli]: https://github.com/mozilla/http-observatory-cli) +[qualys-ssl]: https://www.ssllabs.com/ssltest/analyze.html +[secure_headers]: https://github.com/twitter/secureheaders +[mdn-csp]: https://developer.mozilla.org/en-US/docs/Web/Security/CSP +[github-eng-csp]: http://githubengineering.com/githubs-csp-journey/ +[dropbox-csp-1]: https://blogs.dropbox.com/tech/2015/09/on-csp-reporting-and-filtering/ +[dropbox-csp-2]: https://blogs.dropbox.com/tech/2015/09/unsafe-inline-and-nonce-deployment/ +[dropbox-csp-3]: https://blogs.dropbox.com/tech/2015/09/csp-the-unexpected-eval/ +[dropbox-csp-4]: https://blogs.dropbox.com/tech/2015/09/csp-third-party-integrations-and-privilege-separation/ +[mdn-sri]: https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity +[github-eng-sri]: http://githubengineering.com/subresource-integrity/ +[sprockets-sri]: https://github.com/rails/sprockets-rails#sri-support +[xss]: https://en.wikipedia.org/wiki/Cross-site_scripting +[scss-style-guide]: scss_styleguide.md +[requirements]: ../install/requirements.md#supported-web-browsers + +## Common Errors + +### Rspec (Capybara/Poltergeist) chokes on general JavaScript errors + +If you see very generic JavaScript errors (e.g. `jQuery is undefined`) being thrown in tests, but +can't reproduce them manually, you may have included `ES6`-style JavaScript in files that don't +have the `.js.es6` file extension. Either use ES5-friendly JavaScript or rename the file you're +working in (`git mv <file>.js> <file.js.es6>`). + + diff --git a/doc/development/licensing.md b/doc/development/licensing.md index 8c8c7486fff..05972b33fdb 100644 --- a/doc/development/licensing.md +++ b/doc/development/licensing.md @@ -54,6 +54,7 @@ Libraries with the following licenses are acceptable for use: - [BSD 2-Clause License][BSD-2-Clause]: A permissive (non-copyleft) license as defined by the Open Source Initiative. - [BSD 3-Clause License][BSD-3-Clause] (also known as New BSD or Modified BSD): A permissive (non-copyleft) license as defined by the Open Source Initiative - [ISC License][ISC] (also known as the OpenBSD License): A permissive (non-copyleft) license as defined by the Open Source Initiative. +- [Creative Commons Zero (CC0)][CC0]: A public domain dedication, recommended as a way to disclaim copyright on your work to the maximum extent possible. ## Unacceptable Licenses @@ -85,6 +86,7 @@ Gems which are included only in the "development" or "test" groups by Bundler ar [BSD-2-Clause]: https://opensource.org/licenses/BSD-2-Clause [BSD-3-Clause]: https://opensource.org/licenses/BSD-3-Clause [ISC]: https://opensource.org/licenses/ISC +[CC0]: https://creativecommons.org/publicdomain/zero/1.0/ [GPL]: http://choosealicense.com/licenses/gpl-3.0/ [GPLv2]: http://www.gnu.org/licenses/gpl-2.0.txt [GPLv3]: http://www.gnu.org/licenses/gpl-3.0.txt diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 295eae0a88e..61b0fbc89c9 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -9,10 +9,10 @@ a big burden for most organizations. For this reason it is important that your migrations are written carefully, can be applied online and adhere to the style guide below. Migrations should not require GitLab installations to be taken offline unless -_absolutely_ necessary. If a migration requires downtime this should be -clearly mentioned during the review process as well as being documented in the -monthly release post. For more information see the "Downtime Tagging" section -below. +_absolutely_ necessary - see the ["What Requires Downtime?"](what_requires_downtime.md) +page. If a migration requires downtime, this should be clearly mentioned during +the review process, as well as being documented in the monthly release post. For +more information, see the "Downtime Tagging" section below. When writing your migrations, also consider that databases might have stale data or inconsistencies and guard for that. Try to make as little assumptions as possible diff --git a/doc/development/performance.md b/doc/development/performance.md index 7ff603e2c4a..8337c2d9cb3 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -34,10 +34,11 @@ graphs/dashboards. ## Tooling -GitLab provides two built-in tools to aid the process of improving performance: +GitLab provides built-in tools to aid the process of improving performance: * [Sherlock](profiling.md#sherlock) -* [GitLab Performance Monitoring](../monitoring/performance/monitoring.md) +* [GitLab Performance Monitoring](../administration/monitoring/performance/monitoring.md) +* [Request Profiling](../administration/monitoring/performance/request_profiling.md) GitLab employees can use GitLab.com's performance monitoring systems located at <http://performance.gitlab.net>, this requires you to log in using your @@ -253,5 +254,5 @@ impact on runtime performance, and as such, using a constant instead of referencing an object directly may even slow code down. [#15607]: https://gitlab.com/gitlab-org/gitlab-ce/issues/15607 -[yorickpeterse]: https://gitlab.com/u/yorickpeterse +[yorickpeterse]: https://gitlab.com/yorickpeterse [anti-pattern]: https://en.wikipedia.org/wiki/Anti-pattern diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md new file mode 100644 index 00000000000..e3a20f29a09 --- /dev/null +++ b/doc/development/sidekiq_style_guide.md @@ -0,0 +1,38 @@ +# Sidekiq Style Guide + +This document outlines various guidelines that should be followed when adding or +modifying Sidekiq workers. + +## Default Queue + +Use of the "default" queue is not allowed. Every worker should use a queue that +matches the worker's purpose the closest. For example, workers that are to be +executed periodically should use the "cronjob" queue. + +A list of all available queues can be found in `config/sidekiq_queues.yml`. + +## Dedicated Queues + +Most workers should use their own queue. To ease this process a worker can +include the `DedicatedSidekiqQueue` concern as follows: + +```ruby +class ProcessSomethingWorker + include Sidekiq::Worker + include DedicatedSidekiqQueue +end +``` + +This will set the queue name based on the class' name, minus the `Worker` +suffix. In the above example this would lead to the queue being +`process_something`. + +In some cases multiple workers do use the same queue. For example, the various +workers for updating CI pipelines all use the `pipeline` queue. Adding workers +to existing queues should be done with care, as adding more workers can lead to +slow jobs blocking work (even for different jobs) on the shared queue. + +## Tests + +Each Sidekiq worker must be tested using RSpec, just like any other class. These +tests should be placed in `spec/workers`. diff --git a/doc/gitlab-basics/README.md b/doc/gitlab-basics/README.md index 3aa83975ace..d7e3aa35bdd 100644 --- a/doc/gitlab-basics/README.md +++ b/doc/gitlab-basics/README.md @@ -2,14 +2,14 @@ Step-by-step guides on the basics of working with Git and GitLab. +- [Command line basics](command-line-commands.md) - [Start using Git on the command line](start-using-git.md) - [Create and add your SSH Keys](create-your-ssh-keys.md) -- [Command Line basics](command-line-commands.md) - [Create a project](create-project.md) - [Create a group](create-group.md) - [Create a branch](create-branch.md) - [Fork a project](fork-project.md) - [Add a file](add-file.md) - [Add an image](add-image.md) -- [Create a Merge Request](add-merge-request.md) -- [Create an Issue](create-issue.md) +- [Create an issue](create-issue.md) +- [Create a merge request](add-merge-request.md) diff --git a/doc/gitlab-basics/add-file.md b/doc/gitlab-basics/add-file.md index ff10a98e8f5..e9fbcbc23a9 100644 --- a/doc/gitlab-basics/add-file.md +++ b/doc/gitlab-basics/add-file.md @@ -1,27 +1,5 @@ # How to add a file -You can create a file in your [shell](command-line-commands.md) or in GitLab. - -To create a file in GitLab, sign in to GitLab. - -Select a project on the right side of your screen: - - - -It's a good idea to [create a branch](create-branch.md), but it's not necessary. - -Go to the directory where you'd like to add the file and click on the "+" sign next to the name of the project and directory: - - - -Name your file (you can't add spaces, so you can use hyphens or underscores). Don't forget to include the markup language you'd like to use : - - - -Add all the information that you'd like to include in your file: - - - -Add a commit message based on what you just added and then click on "commit changes": - - +You can create a file in your [terminal](command-line-commands.md) and push +to GitLab or you can use the +[web interface](../user/project/repository/web_editor.md#create-a-file). diff --git a/doc/gitlab-basics/add-merge-request.md b/doc/gitlab-basics/add-merge-request.md index 236b4248ea2..bf01fe51dc3 100644 --- a/doc/gitlab-basics/add-merge-request.md +++ b/doc/gitlab-basics/add-merge-request.md @@ -1,42 +1,33 @@ # How to create a merge request -Merge Requests are useful to integrate separate changes that you've made to a project, on different branches. +Merge requests are useful to integrate separate changes that you've made to a +project, on different branches. This is a brief guide on how to create a merge +request. For more information, check the +[merge requests documentation](../user/project/merge_requests.md). -To create a new Merge Request, sign in to GitLab. +--- -Go to the project where you'd like to merge your changes: +1. Before you start, you should have already [created a branch](create-branch.md) + and [pushed your changes](basic-git-commands.md) to GitLab. - +1. You can then go to the project where you'd like to merge your changes and + click on the **Merge requests** tab. -Click on "Merge Requests" on the left side of your screen: +  - +1. Click on **New merge request** on the right side of the screen. -Click on "+ new Merge Request" on the right side of the screen: +  - +1. Select a source branch and click on the **Compare branches and continue** button. -Select a source branch or branch: +  - +1. At a minimum, add a title and a description to your merge request. Optionally, + select a user to review your merge request and to accept or close it. You may + also select a milestone and labels. -Click on the "compare branches" button: +  - - -Add a title and a description to your Merge Request: - - - -Select a user to review your Merge Request and to accept or close it. You may also select milestones and labels (they are optional). Then click on the "submit new Merge Request" button: - - - -Your Merge Request will be ready to be approved and published. - -### Note - -After you created a new branch, you'll immediately find a "create a Merge Request" button at the top of your screen. -You may automatically create a Merge Request from your recently created branch when clicking on this button: - - +1. When ready, click on the **Submit merge request** button. Your merge request + will be ready to be approved and published. diff --git a/doc/gitlab-basics/basicsimages/add_new_merge_request.png b/doc/gitlab-basics/basicsimages/add_new_merge_request.png Binary files differdeleted file mode 100644 index e60992c4c6a..00000000000 --- a/doc/gitlab-basics/basicsimages/add_new_merge_request.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/add_sshkey.png b/doc/gitlab-basics/basicsimages/add_sshkey.png Binary files differdeleted file mode 100644 index 89c86018629..00000000000 --- a/doc/gitlab-basics/basicsimages/add_sshkey.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/branch_info.png b/doc/gitlab-basics/basicsimages/branch_info.png Binary files differdeleted file mode 100644 index 2264f3c5bf2..00000000000 --- a/doc/gitlab-basics/basicsimages/branch_info.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/branch_name.png b/doc/gitlab-basics/basicsimages/branch_name.png Binary files differdeleted file mode 100644 index 75fe8313611..00000000000 --- a/doc/gitlab-basics/basicsimages/branch_name.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/branches.png b/doc/gitlab-basics/basicsimages/branches.png Binary files differdeleted file mode 100644 index 8621bc05776..00000000000 --- a/doc/gitlab-basics/basicsimages/branches.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/button-create-mr.png b/doc/gitlab-basics/basicsimages/button-create-mr.png Binary files differdeleted file mode 100644 index b52ab148839..00000000000 --- a/doc/gitlab-basics/basicsimages/button-create-mr.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/click-on-new-group.png b/doc/gitlab-basics/basicsimages/click-on-new-group.png Binary files differdeleted file mode 100644 index 6450deec6fc..00000000000 --- a/doc/gitlab-basics/basicsimages/click-on-new-group.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/commit_changes.png b/doc/gitlab-basics/basicsimages/commit_changes.png Binary files differdeleted file mode 100644 index a88809c5a3f..00000000000 --- a/doc/gitlab-basics/basicsimages/commit_changes.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/commit_message.png b/doc/gitlab-basics/basicsimages/commit_message.png Binary files differdeleted file mode 100644 index 4abe4517f98..00000000000 --- a/doc/gitlab-basics/basicsimages/commit_message.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/commits.png b/doc/gitlab-basics/basicsimages/commits.png Binary files differdeleted file mode 100644 index 2bfcaf75f01..00000000000 --- a/doc/gitlab-basics/basicsimages/commits.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/compare_branches.png b/doc/gitlab-basics/basicsimages/compare_branches.png Binary files differdeleted file mode 100644 index 8a18453dd05..00000000000 --- a/doc/gitlab-basics/basicsimages/compare_branches.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/create_file.png b/doc/gitlab-basics/basicsimages/create_file.png Binary files differdeleted file mode 100644 index 5ebe1b227dd..00000000000 --- a/doc/gitlab-basics/basicsimages/create_file.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/create_group.png b/doc/gitlab-basics/basicsimages/create_group.png Binary files differdeleted file mode 100644 index 7ecc3baa990..00000000000 --- a/doc/gitlab-basics/basicsimages/create_group.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/edit_file.png b/doc/gitlab-basics/basicsimages/edit_file.png Binary files differdeleted file mode 100644 index 9d3e817d036..00000000000 --- a/doc/gitlab-basics/basicsimages/edit_file.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/file_located.png b/doc/gitlab-basics/basicsimages/file_located.png Binary files differdeleted file mode 100644 index e357cb5c6ab..00000000000 --- a/doc/gitlab-basics/basicsimages/file_located.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/file_name.png b/doc/gitlab-basics/basicsimages/file_name.png Binary files differdeleted file mode 100644 index 01639c77d0d..00000000000 --- a/doc/gitlab-basics/basicsimages/file_name.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/find_file.png b/doc/gitlab-basics/basicsimages/find_file.png Binary files differdeleted file mode 100644 index 6f26d26ae18..00000000000 --- a/doc/gitlab-basics/basicsimages/find_file.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/find_group.png b/doc/gitlab-basics/basicsimages/find_group.png Binary files differdeleted file mode 100644 index 1211510aae9..00000000000 --- a/doc/gitlab-basics/basicsimages/find_group.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/fork.png b/doc/gitlab-basics/basicsimages/fork.png Binary files differdeleted file mode 100644 index 13ff8345616..00000000000 --- a/doc/gitlab-basics/basicsimages/fork.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/group_info.png b/doc/gitlab-basics/basicsimages/group_info.png Binary files differdeleted file mode 100644 index 2507d6c295b..00000000000 --- a/doc/gitlab-basics/basicsimages/group_info.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/groups.png b/doc/gitlab-basics/basicsimages/groups.png Binary files differdeleted file mode 100644 index ef3dca60cc8..00000000000 --- a/doc/gitlab-basics/basicsimages/groups.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/https.png b/doc/gitlab-basics/basicsimages/https.png Binary files differdeleted file mode 100644 index e74dbc13f9a..00000000000 --- a/doc/gitlab-basics/basicsimages/https.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/image_file.png b/doc/gitlab-basics/basicsimages/image_file.png Binary files differdeleted file mode 100644 index 7f304b8e1f2..00000000000 --- a/doc/gitlab-basics/basicsimages/image_file.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/issue_title.png b/doc/gitlab-basics/basicsimages/issue_title.png Binary files differdeleted file mode 100644 index 60a6f7973be..00000000000 --- a/doc/gitlab-basics/basicsimages/issue_title.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/issues.png b/doc/gitlab-basics/basicsimages/issues.png Binary files differdeleted file mode 100644 index 14e9cdb64e1..00000000000 --- a/doc/gitlab-basics/basicsimages/issues.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/key.png b/doc/gitlab-basics/basicsimages/key.png Binary files differdeleted file mode 100644 index 04400173ce8..00000000000 --- a/doc/gitlab-basics/basicsimages/key.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/merge_requests.png b/doc/gitlab-basics/basicsimages/merge_requests.png Binary files differdeleted file mode 100644 index 570164df18b..00000000000 --- a/doc/gitlab-basics/basicsimages/merge_requests.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/new_issue.png b/doc/gitlab-basics/basicsimages/new_issue.png Binary files differdeleted file mode 100644 index 94e7503dd8b..00000000000 --- a/doc/gitlab-basics/basicsimages/new_issue.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/new_merge_request.png b/doc/gitlab-basics/basicsimages/new_merge_request.png Binary files differdeleted file mode 100644 index 842f5ebed74..00000000000 --- a/doc/gitlab-basics/basicsimages/new_merge_request.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/new_project.png b/doc/gitlab-basics/basicsimages/new_project.png Binary files differdeleted file mode 100644 index 421e8bc247b..00000000000 --- a/doc/gitlab-basics/basicsimages/new_project.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/newbranch.png b/doc/gitlab-basics/basicsimages/newbranch.png Binary files differdeleted file mode 100644 index d5fcf33c4ea..00000000000 --- a/doc/gitlab-basics/basicsimages/newbranch.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/paste_sshkey.png b/doc/gitlab-basics/basicsimages/paste_sshkey.png Binary files differdeleted file mode 100644 index 578ebee4440..00000000000 --- a/doc/gitlab-basics/basicsimages/paste_sshkey.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/profile_settings.png b/doc/gitlab-basics/basicsimages/profile_settings.png Binary files differdeleted file mode 100644 index cb3f79f1879..00000000000 --- a/doc/gitlab-basics/basicsimages/profile_settings.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/project_info.png b/doc/gitlab-basics/basicsimages/project_info.png Binary files differdeleted file mode 100644 index e1adb8d48c2..00000000000 --- a/doc/gitlab-basics/basicsimages/project_info.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/select-group.png b/doc/gitlab-basics/basicsimages/select-group.png Binary files differdeleted file mode 100644 index 33b978dd899..00000000000 --- a/doc/gitlab-basics/basicsimages/select-group.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/select-group2.png b/doc/gitlab-basics/basicsimages/select-group2.png Binary files differdeleted file mode 100644 index aee22c638db..00000000000 --- a/doc/gitlab-basics/basicsimages/select-group2.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/select_branch.png b/doc/gitlab-basics/basicsimages/select_branch.png Binary files differdeleted file mode 100644 index f72a3ffb57f..00000000000 --- a/doc/gitlab-basics/basicsimages/select_branch.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/select_project.png b/doc/gitlab-basics/basicsimages/select_project.png Binary files differdeleted file mode 100644 index 3bb832ea8d0..00000000000 --- a/doc/gitlab-basics/basicsimages/select_project.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/settings.png b/doc/gitlab-basics/basicsimages/settings.png Binary files differdeleted file mode 100644 index 78637013d9b..00000000000 --- a/doc/gitlab-basics/basicsimages/settings.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/shh_keys.png b/doc/gitlab-basics/basicsimages/shh_keys.png Binary files differdeleted file mode 100644 index c87f11a9d3d..00000000000 --- a/doc/gitlab-basics/basicsimages/shh_keys.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/submit_new_issue.png b/doc/gitlab-basics/basicsimages/submit_new_issue.png Binary files differdeleted file mode 100644 index 78b854c8903..00000000000 --- a/doc/gitlab-basics/basicsimages/submit_new_issue.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/title_description_mr.png b/doc/gitlab-basics/basicsimages/title_description_mr.png Binary files differdeleted file mode 100644 index c31d61ec336..00000000000 --- a/doc/gitlab-basics/basicsimages/title_description_mr.png +++ /dev/null diff --git a/doc/gitlab-basics/basicsimages/white_space.png b/doc/gitlab-basics/basicsimages/white_space.png Binary files differdeleted file mode 100644 index eaa969bdcf4..00000000000 --- a/doc/gitlab-basics/basicsimages/white_space.png +++ /dev/null diff --git a/doc/gitlab-basics/command-line-commands.md b/doc/gitlab-basics/command-line-commands.md index addd3b6b6eb..3b075ff5fc0 100644 --- a/doc/gitlab-basics/command-line-commands.md +++ b/doc/gitlab-basics/command-line-commands.md @@ -4,18 +4,21 @@ In Git, when you copy a project you say you "clone" it. To work on a git project locally (from your own computer), you will need to clone it. To do this, sign in to GitLab. -When you are on your Dashboard, click on the project that you'd like to clone, which you'll find at the right side of your screen. +When you are on your Dashboard, click on the project that you'd like to clone. +To work in the project, you can copy a link to the Git repository through a SSH +or a HTTPS protocol. SSH is easier to use after it's been +[setup](create-your-ssh-keys.md). While you are at the **Project** tab, select +HTTPS or SSH from the dropdown menu and copy the link using the 'Copy to clipboard' +button (you'll have to paste it on your shell in the next step). - - -To work in the project, you can copy a link to the Git repository through a SSH or a HTTPS protocol. SSH is easier to use after it's been [setup](create-your-ssh-keys.md). When you're in the project, click on the HTTPS or SSH button at the right side of your screen. Then copy the link (you'll have to paste it on your shell in the next step). - - + ## On the command line ### Clone your project + Go to your computer's shell and type the following command: + ``` git clone PASTE HTTPS OR SSH HERE ``` @@ -23,26 +26,31 @@ git clone PASTE HTTPS OR SSH HERE A clone of the project will be created in your computer. ### Go into a project, directory or file to work in it + ``` cd NAME-OF-PROJECT-OR-FILE ``` ### Go back one directory or file + ``` cd ../ ``` ### View what’s in the directory that you are in + ``` ls ``` ### Create a directory + ``` mkdir NAME-OF-YOUR-DIRECTORY ``` ### Create a README.md or file in directory + ``` touch README.md nano README.md @@ -53,27 +61,33 @@ nano README.md ``` ### Remove a file + ``` rm NAME-OF-FILE ``` ### Remove a directory and all of its contents + ``` rm -rf NAME-OF-DIRECTORY ``` ### View history in the command line + ``` history ``` ### Carry out commands for which the account you are using lacks authority + You will be asked for an administrator’s password. + ``` sudo ``` ### Tell where you are + ``` pwd ``` diff --git a/doc/gitlab-basics/create-branch.md b/doc/gitlab-basics/create-branch.md index 7556b0f663e..ad94f0dad29 100644 --- a/doc/gitlab-basics/create-branch.md +++ b/doc/gitlab-basics/create-branch.md @@ -2,38 +2,11 @@ A branch is an independent line of development. -New commits are recorded in the history for the current branch, which results in taking the source from someone’s repository (the place where the history of your work is stored) at certain point in time, and apply your own changes to it in the history of the project. - -To add changes to your GitLab project, you should create a branch. You can do it in your [shell](basic-git-commands.md) or in GitLab. - -To create a new branch in GitLab, sign in and then select a project on the right side of your screen: - - - -Click on "commits" on the menu on the left side of your screen: - - - -Click on the "branches" tab: - - - -Click on the "new branch" button on the right side of the screen: - - - -Fill out the information required: - -1. Add a name for your new branch (you can't add spaces, so you can use hyphens or underscores) - -1. On the "create from" space, add the the name of the branch you want to branch off from - -1. Click on the button "create branch" - - - -### Note: - -You will be able to find and select the name of your branch in the white box next to a project's name: - - +New commits are recorded in the history for the current branch, which results +in taking the source from someone’s repository (the place where the history of +your work is stored) at certain point in time, and apply your own changes to it +in the history of the project. + +To add changes to your GitLab project, you should create a branch. You can do +it in your [terminal](basic-git-commands.md) or by +[using the web interface](../user/project/repository/web_editor.md#create-a-new-branch). diff --git a/doc/gitlab-basics/create-group.md b/doc/gitlab-basics/create-group.md index f80ae62e442..64274ccd5eb 100644 --- a/doc/gitlab-basics/create-group.md +++ b/doc/gitlab-basics/create-group.md @@ -1,43 +1,48 @@ # How to create a group in GitLab -## Create a group - Your projects in GitLab can be organized in 2 different ways: -under your own namespace for single projects, such as ´your-name/project-1'; or under groups. -If you organize your projects under a group, it works like a folder. You can manage your group members' permissions and access to the projects. - -To create a group, follow the instructions below: +under your own namespace for single projects, such as `your-name/project-1` or +under groups. -Sign in to [GitLab.com](https://gitlab.com). +If you organize your projects under a group, it works like a folder. You can +manage your group members' permissions and access to the projects. -When you are on your Dashboard, click on "Groups" on the left menu of your screen: +--- - +To create a group: -Click on "New group" on the top right side of your screen: +1. Expand the left sidebar by clicking the three bars at the upper left corner + and then navigate to **Groups**. - +  -Fill out the information required: +1. Once in your groups dashboard, click on **New group**. -1. Add a group path or group name (you can't add spaces, so you can use hyphens or underscores) +  -1. Add details or a group description +1. Fill out the needed information: -1. You can choose a group avatar if you'd like + 1. Set the "Group path" which will be the namespace under which your projects + will be hosted (path can contain only letters, digits, underscores, dashes + and dots; it cannot start with dashes or end in dot). + 1. Optionally, you can add a description so that others can briefly understand + what this group is about. + 1. Optionally, choose and avatar for your project. + 1. Choose the [visibility level](../public_access/public_access.md). -1. Click on "create group" +1. Finally, click the **Create group** button. - - -## Add a project to a group +## Add a new project to a group There are 2 different ways to add a new project to a group: -* Select a group and then click on "New project" on the right side of your screen. Then you can [create a project](create-project.md) +- Select a group and then click on the **New project** button. + +  - + You can then continue on [creating a project](create-project.md). -* When you are [creating a project](create-project.md), click on "create a group" on the bottom right side of your screen +- While you are [creating a project](create-project.md), select a group namespace + you've already created from the dropdown menu. - +  diff --git a/doc/gitlab-basics/create-issue.md b/doc/gitlab-basics/create-issue.md index da9a165b8f5..13e5a738c89 100644 --- a/doc/gitlab-basics/create-issue.md +++ b/doc/gitlab-basics/create-issue.md @@ -1,27 +1,30 @@ # How to create an Issue in GitLab -The Issue Tracker is a good place to add things that need to be improved or solved in a project. +The issue tracker is a good place to add things that need to be improved or +solved in a project. -To create an Issue, sign in to GitLab. +--- -Go to the project where you'd like to create the Issue: +1. Go to the project where you'd like to create the issue and navigate to the + **Issues** tab on top. - +  -Click on "Issues" on the left side of your screen: +1. Click on the **New issue** button on the right side of your screen. - +  -Click on the "+ new issue" button on the right side of your screen: +1. At the very minimum, add a title and a description to your issue. + You may assign it to a user, add a milestone or add labels (all optional). - +  -Add a title and a description to your issue: +1. When ready, click on **Submit issue**. - +--- -You may assign the Issue to a user, add a milestone and add labels (they are all optional). Then click on "submit new issue": - - - -Your Issue will now be added to the Issue Tracker and will be ready to be reviewed. You can comment on it and mention the people involved. You can also link Issues to the Merge Requests where the Issues are solved. To do this, you can use an [Issue closing pattern](../user/project/issues/automatic_issue_closing.md). +Your Issue will now be added to the issue tracker of the project you opened it +at and will be ready to be reviewed. You can comment on it and mention the +people involved. You can also link issues to the merge requests where the issues +are solved. To do this, you can use an +[issue closing pattern](../user/project/issues/automatic_issue_closing.md). diff --git a/doc/gitlab-basics/create-project.md b/doc/gitlab-basics/create-project.md index f737dffc024..3f45a631b3a 100644 --- a/doc/gitlab-basics/create-project.md +++ b/doc/gitlab-basics/create-project.md @@ -1,21 +1,24 @@ # How to create a project in GitLab -To create a new project, sign in to GitLab. +There are two ways to create a new project in GitLab. -Go to your Dashboard and click on "new project" on the right side of your screen. +1. While in your dashboard, you can create a new project using the **New project** + green button or you can use the cross icon in the upper right corner next to + your avatar which is always visible. - +  -Fill out the required information: +1. From there you can see several options. -1. Project path or the name of your project (you can't add spaces, so you can use hyphens or underscores) +  -1. Your project's description +1. Fill out the information: -1. Select a [visibility level](https://gitlab.com/help/public_access/public_access) + 1. "Project name" is the name of your project (you can't use spaces, but you + can use hyphens or underscores). + 1. The "Project description" is optional and will be shown in your project's + dashboard so others can briefly understand what your project is about. + 1. Select a [visibility level](../public_access/public_access.md). + 1. You can also [import your existing projects](../workflow/importing/README.md). -1. You can also [import your existing projects](http://docs.gitlab.com/ce/workflow/importing/README.html) - -1. Click on "create project" - -! +1. Finally, click **Create project**. diff --git a/doc/gitlab-basics/create-your-ssh-keys.md b/doc/gitlab-basics/create-your-ssh-keys.md index f31c353f2cf..b6ebe374de3 100644 --- a/doc/gitlab-basics/create-your-ssh-keys.md +++ b/doc/gitlab-basics/create-your-ssh-keys.md @@ -1,33 +1,37 @@ # How to create your SSH Keys -You need to connect your computer to your GitLab account through SSH Keys. They are unique for every computer that you link your GitLab account with. +1. The first thing you need to do is go to your [command line](start-using-git.md) + and follow the [instructions](../ssh/README.md) to generate your SSH key pair. -## Generate your SSH Key +1. Once you do that, login to GitLab with your credentials. +1. On the upper right corner, click on your avatar and go to your **Profile settings**. -Create an account on GitLab. Sign up and check your email for your confirmation link. +  -After you confirm, go to GitLab and sign in to your account. +1. Navigate to the **SSH keys** tab. -## Add your SSH Key +  -On the left side menu, click on "profile settings" and then click on "SSH Keys": +3. Paste your **public** key that you generated in the first step in the 'Key' + box. - +  -Then click on the green button "Add SSH Key": +1. Optionally, give it a descriptive title so that you can recognize it in the + event you add multiple keys. - +  -There, you should paste the SSH Key that your command line will generate for you. Below you'll find the steps to generate it: +1. Finally, click on **Add key** to add it to GitLab. You will be able to see + its fingerprint, its title and creation date. - +  -## To generate an SSH Key on your command line -Go to your [command line](start-using-git.md) and follow the [instructions](../ssh/README.md) to generate it. +>**Note:** +Once you add a key, you cannot edit it, only remove it. In case the paste +didn't work, you will have to remove the offending key and re-add it. -Copy the SSH Key that your command line created and paste it on the "Key" box on the GitLab page. The title will be added automatically. +--- - - -Now, you'll be able to use Git over SSH, instead of Git over HTTP. +Congratulations! You are now ready to use Git over SSH, instead of Git over HTTP! diff --git a/doc/gitlab-basics/fork-project.md b/doc/gitlab-basics/fork-project.md index 5f8b81ea919..6c232fe6086 100644 --- a/doc/gitlab-basics/fork-project.md +++ b/doc/gitlab-basics/fork-project.md @@ -1,19 +1,20 @@ # How to fork a project -A fork is a copy of an original repository that you can put somewhere else -or where you can experiment and apply changes that you can later decide if +A fork is a copy of an original repository that you can put in another namespace +where you can experiment and apply changes that you can later decide if publishing or not, without affecting your original project. It takes just a few steps to fork a project in GitLab. -Sign in to GitLab. +1. Go to a project's dashboard under the **Project** tab and click on the + **Fork** button. -Select a project on the right side of your screen: +  - +1. You will be asked where to fork the repository. Click on the user or group + to where you'd like to add the forked project. -Click on the "fork" button on the right side of your screen: +  - - -Click on the user or group to where you'd like to add the forked project. +1. After a few moments, depending on the repository's size, the forking will + complete. diff --git a/doc/gitlab-basics/img/create_new_group_info.png b/doc/gitlab-basics/img/create_new_group_info.png Binary files differnew file mode 100644 index 00000000000..c8eddfd1bbb --- /dev/null +++ b/doc/gitlab-basics/img/create_new_group_info.png diff --git a/doc/gitlab-basics/img/create_new_group_sidebar.png b/doc/gitlab-basics/img/create_new_group_sidebar.png Binary files differnew file mode 100644 index 00000000000..28017ee02e0 --- /dev/null +++ b/doc/gitlab-basics/img/create_new_group_sidebar.png diff --git a/doc/gitlab-basics/img/create_new_project_button.png b/doc/gitlab-basics/img/create_new_project_button.png Binary files differnew file mode 100644 index 00000000000..e7c794d943f --- /dev/null +++ b/doc/gitlab-basics/img/create_new_project_button.png diff --git a/doc/gitlab-basics/img/create_new_project_from_group.png b/doc/gitlab-basics/img/create_new_project_from_group.png Binary files differnew file mode 100644 index 00000000000..6d41d17f9ca --- /dev/null +++ b/doc/gitlab-basics/img/create_new_project_from_group.png diff --git a/doc/gitlab-basics/img/create_new_project_info.png b/doc/gitlab-basics/img/create_new_project_info.png Binary files differnew file mode 100644 index 00000000000..16d56f0707f --- /dev/null +++ b/doc/gitlab-basics/img/create_new_project_info.png diff --git a/doc/gitlab-basics/img/fork_choose_namespace.png b/doc/gitlab-basics/img/fork_choose_namespace.png Binary files differnew file mode 100644 index 00000000000..82c9c3bd39e --- /dev/null +++ b/doc/gitlab-basics/img/fork_choose_namespace.png diff --git a/doc/gitlab-basics/img/fork_new.png b/doc/gitlab-basics/img/fork_new.png Binary files differnew file mode 100644 index 00000000000..41885223286 --- /dev/null +++ b/doc/gitlab-basics/img/fork_new.png diff --git a/doc/gitlab-basics/img/merge_request_new.png b/doc/gitlab-basics/img/merge_request_new.png Binary files differnew file mode 100644 index 00000000000..0aba5743f01 --- /dev/null +++ b/doc/gitlab-basics/img/merge_request_new.png diff --git a/doc/gitlab-basics/img/merge_request_page.png b/doc/gitlab-basics/img/merge_request_page.png Binary files differnew file mode 100644 index 00000000000..68c3bbf9444 --- /dev/null +++ b/doc/gitlab-basics/img/merge_request_page.png diff --git a/doc/gitlab-basics/img/merge_request_select_branch.png b/doc/gitlab-basics/img/merge_request_select_branch.png Binary files differnew file mode 100644 index 00000000000..516436ff6cc --- /dev/null +++ b/doc/gitlab-basics/img/merge_request_select_branch.png diff --git a/doc/gitlab-basics/img/new_issue_button.png b/doc/gitlab-basics/img/new_issue_button.png Binary files differnew file mode 100644 index 00000000000..46b626bed65 --- /dev/null +++ b/doc/gitlab-basics/img/new_issue_button.png diff --git a/doc/gitlab-basics/img/new_issue_page.png b/doc/gitlab-basics/img/new_issue_page.png Binary files differnew file mode 100644 index 00000000000..843504130b7 --- /dev/null +++ b/doc/gitlab-basics/img/new_issue_page.png diff --git a/doc/gitlab-basics/img/profile_settings.png b/doc/gitlab-basics/img/profile_settings.png Binary files differnew file mode 100644 index 00000000000..f0abd478849 --- /dev/null +++ b/doc/gitlab-basics/img/profile_settings.png diff --git a/doc/gitlab-basics/img/profile_settings_ssh_keys.png b/doc/gitlab-basics/img/profile_settings_ssh_keys.png Binary files differnew file mode 100644 index 00000000000..2c9a42fe10c --- /dev/null +++ b/doc/gitlab-basics/img/profile_settings_ssh_keys.png diff --git a/doc/gitlab-basics/img/profile_settings_ssh_keys_paste_pub.png b/doc/gitlab-basics/img/profile_settings_ssh_keys_paste_pub.png Binary files differnew file mode 100644 index 00000000000..cd7add6937f --- /dev/null +++ b/doc/gitlab-basics/img/profile_settings_ssh_keys_paste_pub.png diff --git a/doc/gitlab-basics/img/profile_settings_ssh_keys_single_key.png b/doc/gitlab-basics/img/profile_settings_ssh_keys_single_key.png Binary files differnew file mode 100644 index 00000000000..095beb02be8 --- /dev/null +++ b/doc/gitlab-basics/img/profile_settings_ssh_keys_single_key.png diff --git a/doc/gitlab-basics/img/profile_settings_ssh_keys_title.png b/doc/gitlab-basics/img/profile_settings_ssh_keys_title.png Binary files differnew file mode 100644 index 00000000000..4b998a7f948 --- /dev/null +++ b/doc/gitlab-basics/img/profile_settings_ssh_keys_title.png diff --git a/doc/gitlab-basics/img/project_clone_url.png b/doc/gitlab-basics/img/project_clone_url.png Binary files differnew file mode 100644 index 00000000000..eed430e1036 --- /dev/null +++ b/doc/gitlab-basics/img/project_clone_url.png diff --git a/doc/gitlab-basics/img/project_navbar.png b/doc/gitlab-basics/img/project_navbar.png Binary files differnew file mode 100644 index 00000000000..97cf3cd9702 --- /dev/null +++ b/doc/gitlab-basics/img/project_navbar.png diff --git a/doc/gitlab-basics/basicsimages/public_file_link.png b/doc/gitlab-basics/img/public_file_link.png Binary files differindex f60df6807f4..f60df6807f4 100644 --- a/doc/gitlab-basics/basicsimages/public_file_link.png +++ b/doc/gitlab-basics/img/public_file_link.png diff --git a/doc/gitlab-basics/img/select_group_dropdown.png b/doc/gitlab-basics/img/select_group_dropdown.png Binary files differnew file mode 100644 index 00000000000..7d8b89c2df9 --- /dev/null +++ b/doc/gitlab-basics/img/select_group_dropdown.png diff --git a/doc/gitlab-basics/start-using-git.md b/doc/gitlab-basics/start-using-git.md index b61f436c1a4..42cd8bb3e48 100644 --- a/doc/gitlab-basics/start-using-git.md +++ b/doc/gitlab-basics/start-using-git.md @@ -1,11 +1,10 @@ # Start using Git on the command line -If you want to start using a Git and GitLab, make sure that you have created an -account on GitLab. +If you want to start using Git and GitLab, make sure that you have created and/or signed into an account on GitLab. ## Open a shell -Depending on your operating system, find the shell of your preference. Here are some suggestions. +Depending on your operating system, you will need to use a shell of your preference. Here are some suggestions: - [Terminal](http://blog.teamtreehouse.com/introduction-to-the-mac-os-x-command-line) on Mac OSX @@ -22,19 +21,19 @@ Type the following command and then press enter: git --version ``` -You should receive a message that will tell you which Git version you have in your computer. If you don’t receive a "Git version" message, it means that you need to [download Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git). +You should receive a message that will tell you which Git version you have on your computer. If you don’t receive a "Git version" message, it means that you need to [download Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git). If Git doesn't automatically download, there's an option on the website to [download manually](https://git-scm.com/downloads). Then follow the steps on the installation window. -After you finished installing, open a new shell and type "git --version" again to verify that it was correctly installed. +After you are finished installing, open a new shell and type "git --version" again to verify that it was correctly installed. ## Add your Git username and set your email -It is important because every Git commit that you create will use this information. +It is important to configure your Git username and email address as every Git commit will use this information to identify you as the author. On your shell, type the following command to add your username: ``` -git config --global user.name ADD YOUR USERNAME +git config --global user.name "YOUR_USERNAME" ``` Then verify that you have the correct username: @@ -44,7 +43,7 @@ git config --global user.name To set your email address, type the following command: ``` -git config --global user.email ADD YOUR EMAIL +git config --global user.email "your_email_address@example.com" ``` To verify that you entered your email correctly, type: @@ -52,7 +51,7 @@ To verify that you entered your email correctly, type: git config --global user.email ``` -You'll need to do this only once because you are using the "--global" option. It tells Git to always use this information for anything you do on that system. If you want to override this with a different username or email address for specific projects, you can run the command without the "--global" option when you’re in that project. +You'll need to do this only once as you are using the `--global` option. It tells Git to always use this information for anything you do on that system. If you want to override this with a different username or email address for specific projects, you can run the command without the `--global` option when you’re in that project. ## Check your information @@ -76,7 +75,7 @@ git pull REMOTE NAME-OF-BRANCH -u (REMOTE: origin) (NAME-OF-BRANCH: could be "master" or an existing branch) ### Create a branch -Spaces won't be recognized, so you need to use a hyphen or underscore. +Spaces won't be recognized, so you will need to use a hyphen or underscore. ``` git checkout -b NAME-OF-BRANCH ``` @@ -127,4 +126,3 @@ You need to be in the master branch. git checkout master git merge NAME-OF-BRANCH ``` - diff --git a/doc/incoming_email/README.md b/doc/incoming_email/README.md index 5a9a1582877..db0f03f2c98 100644 --- a/doc/incoming_email/README.md +++ b/doc/incoming_email/README.md @@ -1,302 +1 @@ -# Reply by email - -GitLab can be set up to allow users to comment on issues and merge requests by -replying to notification emails. - -## Requirement - -Reply by email requires an IMAP-enabled email account. GitLab allows you to use -three strategies for this feature: -- using email sub-addressing -- using a dedicated email address -- using a catch-all mailbox - -### Email sub-addressing - -**If your provider or server supports email sub-addressing, we recommend using it.** - -[Sub-addressing](https://en.wikipedia.org/wiki/Email_address#Sub-addressing) is -a feature where any email to `user+some_arbitrary_tag@example.com` will end up -in the mailbox for `user@example.com`, and is supported by providers such as -Gmail, Google Apps, Yahoo! Mail, Outlook.com and iCloud, as well as the Postfix -mail server which you can run on-premises. - -### Dedicated email address - -This solution is really simple to set up: you just have to create an email -address dedicated to receive your users' replies to GitLab notifications. - -### Catch-all mailbox - -A [catch-all mailbox](https://en.wikipedia.org/wiki/Catch-all) for a domain will -"catch all" the emails addressed to the domain that do not exist in the mail -server. - -## How it works? - -### 1. GitLab sends a notification email - -When GitLab sends a notification and Reply by email is enabled, the `Reply-To` -header is set to the address defined in your GitLab configuration, with the -`%{key}` placeholder (if present) replaced by a specific "reply key". In -addition, this "reply key" is also added to the `References` header. - -### 2. You reply to the notification email - -When you reply to the notification email, your email client will: - -- send the email to the `Reply-To` address it got from the notification email -- set the `In-Reply-To` header to the value of the `Message-ID` header from the - notification email -- set the `References` header to the value of the `Message-ID` plus the value of - the notification email's `References` header. - -### 3. GitLab receives your reply to the notification email - -When GitLab receives your reply, it will look for the "reply key" in the -following headers, in this order: - -1. the `To` header -1. the `References` header - -If it finds a reply key, it will be able to leave your reply as a comment on -the entity the notification was about (issue, merge request, commit...). - -For more details about the `Message-ID`, `In-Reply-To`, and `References headers`, -please consult [RFC 5322](https://tools.ietf.org/html/rfc5322#section-3.6.4). - -## Set it up - -If you want to use Gmail / Google Apps with Reply by email, make sure you have -[IMAP access enabled](https://support.google.com/mail/troubleshooter/1668960?hl=en#ts=1665018) -and [allowed less secure apps to access the account](https://support.google.com/accounts/answer/6010255). - -To set up a basic Postfix mail server with IMAP access on Ubuntu, follow -[these instructions](./postfix.md). - -### Omnibus package installations - -1. Find the `incoming_email` section in `/etc/gitlab/gitlab.rb`, enable the - feature and fill in the details for your specific IMAP server and email account: - - ```ruby - # Configuration for Postfix mail server, assumes mailbox incoming@gitlab.example.com - gitlab_rails['incoming_email_enabled'] = true - - # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. - # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). - gitlab_rails['incoming_email_address'] = "incoming+%{key}@gitlab.example.com" - - # Email account username - # With third party providers, this is usually the full email address. - # With self-hosted email servers, this is usually the user part of the email address. - gitlab_rails['incoming_email_email'] = "incoming" - # Email account password - gitlab_rails['incoming_email_password'] = "[REDACTED]" - - # IMAP server host - gitlab_rails['incoming_email_host'] = "gitlab.example.com" - # IMAP server port - gitlab_rails['incoming_email_port'] = 143 - # Whether the IMAP server uses SSL - gitlab_rails['incoming_email_ssl'] = false - # Whether the IMAP server uses StartTLS - gitlab_rails['incoming_email_start_tls'] = false - - # The mailbox where incoming mail will end up. Usually "inbox". - gitlab_rails['incoming_email_mailbox_name'] = "inbox" - ``` - - ```ruby - # Configuration for Gmail / Google Apps, assumes mailbox gitlab-incoming@gmail.com - gitlab_rails['incoming_email_enabled'] = true - - # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. - # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). - gitlab_rails['incoming_email_address'] = "gitlab-incoming+%{key}@gmail.com" - - # Email account username - # With third party providers, this is usually the full email address. - # With self-hosted email servers, this is usually the user part of the email address. - gitlab_rails['incoming_email_email'] = "gitlab-incoming@gmail.com" - # Email account password - gitlab_rails['incoming_email_password'] = "[REDACTED]" - - # IMAP server host - gitlab_rails['incoming_email_host'] = "imap.gmail.com" - # IMAP server port - gitlab_rails['incoming_email_port'] = 993 - # Whether the IMAP server uses SSL - gitlab_rails['incoming_email_ssl'] = true - # Whether the IMAP server uses StartTLS - gitlab_rails['incoming_email_start_tls'] = false - - # The mailbox where incoming mail will end up. Usually "inbox". - gitlab_rails['incoming_email_mailbox_name'] = "inbox" - ``` - -1. Reconfigure GitLab and restart mailroom for the changes to take effect: - - ```sh - sudo gitlab-ctl reconfigure - sudo gitlab-ctl restart mailroom - ``` - -1. Verify that everything is configured correctly: - - ```sh - sudo gitlab-rake gitlab:incoming_email:check - ``` - -1. Reply by email should now be working. - -### Installations from source - -1. Go to the GitLab installation directory: - - ```sh - cd /home/git/gitlab - ``` - -1. Find the `incoming_email` section in `config/gitlab.yml`, enable the feature - and fill in the details for your specific IMAP server and email account: - - ```sh - sudo editor config/gitlab.yml - ``` - - ```yaml - # Configuration for Postfix mail server, assumes mailbox incoming@gitlab.example.com - incoming_email: - enabled: true - - # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. - # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). - address: "incoming+%{key}@gitlab.example.com" - - # Email account username - # With third party providers, this is usually the full email address. - # With self-hosted email servers, this is usually the user part of the email address. - user: "incoming" - # Email account password - password: "[REDACTED]" - - # IMAP server host - host: "gitlab.example.com" - # IMAP server port - port: 143 - # Whether the IMAP server uses SSL - ssl: false - # Whether the IMAP server uses StartTLS - start_tls: false - - # The mailbox where incoming mail will end up. Usually "inbox". - mailbox: "inbox" - ``` - - ```yaml - # Configuration for Gmail / Google Apps, assumes mailbox gitlab-incoming@gmail.com - incoming_email: - enabled: true - - # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. - # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). - address: "gitlab-incoming+%{key}@gmail.com" - - # Email account username - # With third party providers, this is usually the full email address. - # With self-hosted email servers, this is usually the user part of the email address. - user: "gitlab-incoming@gmail.com" - # Email account password - password: "[REDACTED]" - - # IMAP server host - host: "imap.gmail.com" - # IMAP server port - port: 993 - # Whether the IMAP server uses SSL - ssl: true - # Whether the IMAP server uses StartTLS - start_tls: false - - # The mailbox where incoming mail will end up. Usually "inbox". - mailbox: "inbox" - ``` - -1. Enable `mail_room` in the init script at `/etc/default/gitlab`: - - ```sh - sudo mkdir -p /etc/default - echo 'mail_room_enabled=true' | sudo tee -a /etc/default/gitlab - ``` - -1. Restart GitLab: - - ```sh - sudo service gitlab restart - ``` - -1. Verify that everything is configured correctly: - - ```sh - sudo -u git -H bundle exec rake gitlab:incoming_email:check RAILS_ENV=production - ``` - -1. Reply by email should now be working. - -### Development - -1. Go to the GitLab installation directory. - -1. Find the `incoming_email` section in `config/gitlab.yml`, enable the feature and fill in the details for your specific IMAP server and email account: - - ```yaml - # Configuration for Gmail / Google Apps, assumes mailbox gitlab-incoming@gmail.com - incoming_email: - enabled: true - - # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. - # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). - address: "gitlab-incoming+%{key}@gmail.com" - - # Email account username - # With third party providers, this is usually the full email address. - # With self-hosted email servers, this is usually the user part of the email address. - user: "gitlab-incoming@gmail.com" - # Email account password - password: "[REDACTED]" - - # IMAP server host - host: "imap.gmail.com" - # IMAP server port - port: 993 - # Whether the IMAP server uses SSL - ssl: true - # Whether the IMAP server uses StartTLS - start_tls: false - - # The mailbox where incoming mail will end up. Usually "inbox". - mailbox: "inbox" - ``` - - As mentioned, the part after `+` is ignored, and this will end up in the mailbox for `gitlab-incoming@gmail.com`. - -1. Uncomment the `mail_room` line in your `Procfile`: - - ```yaml - mail_room: bundle exec mail_room -q -c config/mail_room.yml - ``` - -1. Restart GitLab: - - ```sh - bundle exec foreman start - ``` - -1. Verify that everything is configured correctly: - - ```sh - bundle exec rake gitlab:incoming_email:check RAILS_ENV=development - ``` - -1. Reply by email should now be working. +This document was moved to [administration/reply_by_email](../administration/reply_by_email.md). diff --git a/doc/incoming_email/postfix.md b/doc/incoming_email/postfix.md index 787d21f7f8f..90833238ac5 100644 --- a/doc/incoming_email/postfix.md +++ b/doc/incoming_email/postfix.md @@ -1,321 +1 @@ -# Set up Postfix for Reply by email - -This document will take you through the steps of setting up a basic Postfix mail server with IMAP authentication on Ubuntu, to be used with Reply by email. - -The instructions make the assumption that you will be using the email address `incoming@gitlab.example.com`, that is, username `incoming` on host `gitlab.example.com`. Don't forget to change it to your actual host when executing the example code snippets. - -## Configure your server firewall - -1. Open up port 25 on your server so that people can send email into the server over SMTP. -2. If the mail server is different from the server running GitLab, open up port 143 on your server so that GitLab can read email from the server over IMAP. - -## Install packages - -1. Install the `postfix` package if it is not installed already: - - ```sh - sudo apt-get install postfix - ``` - - When asked about the environment, select 'Internet Site'. When asked to confirm the hostname, make sure it matches `gitlab.example.com`. - -1. Install the `mailutils` package. - - ```sh - sudo apt-get install mailutils - ``` - -## Create user - -1. Create a user for incoming email. - - ```sh - sudo useradd -m -s /bin/bash incoming - ``` - -1. Set a password for this user. - - ```sh - sudo passwd incoming - ``` - - Be sure not to forget this, you'll need it later. - -## Test the out-of-the-box setup - -1. Connect to the local SMTP server: - - ```sh - telnet localhost 25 - ``` - - You should see a prompt like this: - - ```sh - Trying 127.0.0.1... - Connected to localhost. - Escape character is '^]'. - 220 gitlab.example.com ESMTP Postfix (Ubuntu) - ``` - - If you get a `Connection refused` error instead, verify that `postfix` is running: - - ```sh - sudo postfix status - ``` - - If it is not, start it: - - ```sh - sudo postfix start - ``` - -1. Send the new `incoming` user a dummy email to test SMTP, by entering the following into the SMTP prompt: - - ``` - ehlo localhost - mail from: root@localhost - rcpt to: incoming@localhost - data - Subject: Re: Some issue - - Sounds good! - . - quit - ``` - - _**Note:** The `.` is a literal period on its own line._ - - _**Note:** If you receive an error after entering `rcpt to: incoming@localhost` - then your Postfix `my_network` configuration is not correct. The error will - say 'Temporary lookup failure'. See - [Configure Postfix to receive email from the Internet](#configure-postfix-to-receive-email-from-the-internet)._ - -1. Check if the `incoming` user received the email: - - ```sh - su - incoming - mail - ``` - - You should see output like this: - - ``` - "/var/mail/incoming": 1 message 1 unread - >U 1 root@localhost 59/2842 Re: Some issue - ``` - - Quit the mail app: - - ```sh - q - ``` - -1. Log out of the `incoming` account and go back to being `root`: - - ```sh - logout - ``` - -## Configure Postfix to use Maildir-style mailboxes - -Courier, which we will install later to add IMAP authentication, requires mailboxes to have the Maildir format, rather than mbox. - -1. Configure Postfix to use Maildir-style mailboxes: - - ```sh - sudo postconf -e "home_mailbox = Maildir/" - ``` - -1. Restart Postfix: - - ```sh - sudo /etc/init.d/postfix restart - ``` - -1. Test the new setup: - - 1. Follow steps 1 and 2 of _[Test the out-of-the-box setup](#test-the-out-of-the-box-setup)_. - 1. Check if the `incoming` user received the email: - - ```sh - su - incoming - MAIL=/home/incoming/Maildir - mail - ``` - - You should see output like this: - - ``` - "/home/incoming/Maildir": 1 message 1 unread - >U 1 root@localhost 59/2842 Re: Some issue - ``` - - Quit the mail app: - - ```sh - q - ``` - - _**Note:** If `mail` returns an error `Maildir: Is a directory` then your - version of `mail` doesn't support Maildir style mailboxes. Install - `heirloom-mailx` by running `sudo apt-get install heirloom-mailx`. Then, - try the above steps again, substituting `heirloom-mailx` for the `mail` - command._ - -1. Log out of the `incoming` account and go back to being `root`: - - ```sh - logout - ``` - -## Install the Courier IMAP server - -1. Install the `courier-imap` package: - - ```sh - sudo apt-get install courier-imap - ``` - -## Configure Postfix to receive email from the internet - -1. Let Postfix know about the domains that it should consider local: - - ```sh - sudo postconf -e "mydestination = gitlab.example.com, localhost.localdomain, localhost" - ``` - -1. Let Postfix know about the IPs that it should consider part of the LAN: - - We'll assume `192.168.1.0/24` is your local LAN. You can safely skip this step if you don't have other machines in the same local network. - - ```sh - sudo postconf -e "mynetworks = 127.0.0.0/8, 192.168.1.0/24" - ``` - -1. Configure Postfix to receive mail on all interfaces, which includes the internet: - - ```sh - sudo postconf -e "inet_interfaces = all" - ``` - -1. Configure Postfix to use the `+` delimiter for sub-addressing: - - ```sh - sudo postconf -e "recipient_delimiter = +" - ``` - -1. Restart Postfix: - - ```sh - sudo service postfix restart - ``` - -## Test the final setup - -1. Test SMTP under the new setup: - - 1. Connect to the SMTP server: - - ```sh - telnet gitlab.example.com 25 - ``` - - You should see a prompt like this: - - ```sh - Trying 123.123.123.123... - Connected to gitlab.example.com. - Escape character is '^]'. - 220 gitlab.example.com ESMTP Postfix (Ubuntu) - ``` - - If you get a `Connection refused` error instead, make sure your firewall is setup to allow inbound traffic on port 25. - - 1. Send the `incoming` user a dummy email to test SMTP, by entering the following into the SMTP prompt: - - ``` - ehlo gitlab.example.com - mail from: root@gitlab.example.com - rcpt to: incoming@gitlab.example.com - data - Subject: Re: Some issue - - Sounds good! - . - quit - ``` - - (Note: The `.` is a literal period on its own line) - - 1. Check if the `incoming` user received the email: - - ```sh - su - incoming - MAIL=/home/incoming/Maildir - mail - ``` - - You should see output like this: - - ``` - "/home/incoming/Maildir": 1 message 1 unread - >U 1 root@gitlab.example.com 59/2842 Re: Some issue - ``` - - Quit the mail app: - - ```sh - q - ``` - - 1. Log out of the `incoming` account and go back to being `root`: - - ```sh - logout - ``` - -1. Test IMAP under the new setup: - - 1. Connect to the IMAP server: - - ```sh - telnet gitlab.example.com 143 - ``` - - You should see a prompt like this: - - ```sh - Trying 123.123.123.123... - Connected to mail.example.gitlab.com. - Escape character is '^]'. - - OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION] Courier-IMAP ready. Copyright 1998-2011 Double Precision, Inc. See COPYING for distribution information. - ``` - - 1. Sign in as the `incoming` user to test IMAP, by entering the following into the IMAP prompt: - - ``` - a login incoming PASSWORD - ``` - - Replace PASSWORD with the password you set on the `incoming` user earlier. - - You should see output like this: - - ``` - a OK LOGIN Ok. - ``` - - 1. Disconnect from the IMAP server: - - ```sh - a logout - ``` - -## Done! - -If all the tests were successful, Postfix is all set up and ready to receive email! Continue with the [Reply by email](./README.md) guide to configure GitLab. - ---------- - -_This document was adapted from https://help.ubuntu.com/community/PostfixBasicSetupHowto, by contributors to the Ubuntu documentation wiki._ +This document was moved to [administration/reply_by_email_postfix_setup](../administration/reply_by_email_postfix_setup.md). diff --git a/doc/install/installation.md b/doc/install/installation.md index cb4c1f4a091..c9acc9cdfb0 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -108,7 +108,7 @@ Then select 'Internet Site' and press enter to confirm the hostname. ## 2. Ruby -_**Note:** The current supported Ruby versions are 2.1.x and 2.3.x. 2.3.x is preferred, and support for 2.1.x will be dropped in the future. +**Note:** The current supported Ruby versions are 2.1.x and 2.3.x. 2.3.x is preferred, and support for 2.1.x will be dropped in the future. The use of Ruby version managers such as [RVM], [rbenv] or [chruby] with GitLab in production, frequently leads to hard to diagnose problems. For example, @@ -268,9 +268,9 @@ sudo usermod -aG redis git ### Clone the Source # Clone GitLab repository - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 8-12-stable gitlab + sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 8-13-stable gitlab -**Note:** You can change `8-12-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! +**Note:** You can change `8-13-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! ### Configure It @@ -400,7 +400,7 @@ If you are not using Linux you may have to run `gmake` instead of cd /home/git sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-workhorse.git cd gitlab-workhorse - sudo -u git -H git checkout v0.8.2 + sudo -u git -H git checkout v0.8.5 sudo -u git -H make ### Initialize Database and Activate Advanced Features @@ -563,7 +563,7 @@ Using a self-signed certificate is discouraged but if you must use it follow the ### Enable Reply by email -See the ["Reply by email" documentation](../incoming_email/README.md) for more information on how to set this up. +See the ["Reply by email" documentation](../administration/reply_by_email.md) for more information on how to set this up. ### LDAP Authentication diff --git a/doc/monitoring/health_check.md b/doc/monitoring/health_check.md index eac57bc3de4..6cf93c33ec2 100644 --- a/doc/monitoring/health_check.md +++ b/doc/monitoring/health_check.md @@ -1,66 +1 @@ -# Health Check - -> [Introduced][ce-3888] in GitLab 8.8. - -GitLab provides a health check endpoint for uptime monitoring on the `health_check` web -endpoint. The health check reports on the overall system status based on the status of -the database connection, the state of the database migrations, and the ability to write -and access the cache. This endpoint can be provided to uptime monitoring services like -[Pingdom][pingdom], [Nagios][nagios-health], and [NewRelic][newrelic-health]. - -## Access Token - -An access token needs to be provided while accessing the health check endpoint. The current -accepted token can be found on the `admin/health_check` page of your GitLab instance. - - - -The access token can be passed as a URL parameter: - -``` -https://gitlab.example.com/health_check.json?token=ACCESS_TOKEN -``` - -or as an HTTP header: - -```bash -curl --header "TOKEN: ACCESS_TOKEN" https://gitlab.example.com/health_check.json -``` - -## Using the Endpoint - -Once you have the access token, health information can be retrieved as plain text, JSON, -or XML using the `health_check` endpoint: - -- `https://gitlab.example.com/health_check?token=ACCESS_TOKEN` -- `https://gitlab.example.com/health_check.json?token=ACCESS_TOKEN` -- `https://gitlab.example.com/health_check.xml?token=ACCESS_TOKEN` - -You can also ask for the status of specific services: - -- `https://gitlab.example.com/health_check/cache.json?token=ACCESS_TOKEN` -- `https://gitlab.example.com/health_check/database.json?token=ACCESS_TOKEN` -- `https://gitlab.example.com/health_check/migrations.json?token=ACCESS_TOKEN` - -For example, the JSON output of the following health check: - -```bash -curl --header "TOKEN: ACCESS_TOKEN" https://gitlab.example.com/health_check.json -``` - -would be like: - -``` -{"healthy":true,"message":"success"} -``` - -## Status - -On failure, the endpoint will return a `500` HTTP status code. On success, the endpoint -will return a valid successful HTTP status code, and a `success` message. Ideally your -uptime monitoring should look for the success message. - -[ce-3888]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3888 -[pingdom]: https://www.pingdom.com -[nagios-health]: https://nagios-plugins.org/doc/man/check_http.html -[newrelic-health]: https://docs.newrelic.com/docs/alerts/alert-policies/downtime-alerts/availability-monitoring +This document was moved to [user/admin_area/monitoring/health_check](../user/admin_area/monitoring/health_check.md). diff --git a/doc/monitoring/performance/gitlab_configuration.md b/doc/monitoring/performance/gitlab_configuration.md index 771584268d9..19d46135930 100644 --- a/doc/monitoring/performance/gitlab_configuration.md +++ b/doc/monitoring/performance/gitlab_configuration.md @@ -1,40 +1 @@ -# GitLab Configuration - -GitLab Performance Monitoring is disabled by default. To enable it and change any of its -settings, navigate to the Admin area in **Settings > Metrics** -(`/admin/application_settings`). - -The minimum required settings you need to set are the InfluxDB host and port. -Make sure _Enable InfluxDB Metrics_ is checked and hit **Save** to save the -changes. - ---- - - - ---- - -Finally, a restart of all GitLab processes is required for the changes to take -effect: - -```bash -# For Omnibus installations -sudo gitlab-ctl restart - -# For installations from source -sudo service gitlab restart -``` - -## Pending Migrations - -When any migrations are pending, the metrics are disabled until the migrations -have been performed. - ---- - -Read more on: - -- [Introduction to GitLab Performance Monitoring](introduction.md) -- [InfluxDB Configuration](influxdb_configuration.md) -- [InfluxDB Schema](influxdb_schema.md) -- [Grafana Install/Configuration](grafana_configuration.md) +This document was moved to [administration/monitoring/performance/gitlab_configuration](../../administration/monitoring/performance/gitlab_configuration.md). diff --git a/doc/monitoring/performance/grafana_configuration.md b/doc/monitoring/performance/grafana_configuration.md index 7947b0fedc4..0d4be02ff5f 100644 --- a/doc/monitoring/performance/grafana_configuration.md +++ b/doc/monitoring/performance/grafana_configuration.md @@ -1,111 +1 @@ -# Grafana Configuration - -[Grafana](http://grafana.org/) is a tool that allows you to visualize time -series metrics through graphs and dashboards. It supports several backend -data stores, including InfluxDB. GitLab writes performance data to InfluxDB -and Grafana will allow you to query InfluxDB to display useful graphs. - -For the easiest installation and configuration, install Grafana on the same -server as InfluxDB. For larger installations, you may want to split out these -services. - -## Installation - -Grafana supplies package repositories (Yum/Apt) for easy installation. -See [Grafana installation documentation](http://docs.grafana.org/installation/) -for detailed steps. - -> **Note**: Before starting Grafana for the first time, set the admin user -and password in `/etc/grafana/grafana.ini`. Otherwise, the default password -will be `admin`. - -## Configuration - -Login as the admin user. Expand the menu by clicking the Grafana logo in the -top left corner. Choose 'Data Sources' from the menu. Then, click 'Add new' -in the top bar. - - - -Fill in the configuration details for the InfluxDB data source. Save and -Test Connection to ensure the configuration is correct. - -- **Name**: InfluxDB -- **Default**: Checked -- **Type**: InfluxDB 0.9.x (Even if you're using InfluxDB 0.10.x) -- **Url**: https://localhost:8086 (Or the remote URL if you've installed InfluxDB -on a separate server) -- **Access**: proxy -- **Database**: gitlab -- **User**: admin (Or the username configured when setting up InfluxDB) -- **Password**: The password configured when you set up InfluxDB - - - -## Apply retention policies and create continuous queries - -If you intend to import the GitLab provided Grafana dashboards, you will need to -set up the right retention policies and continuous queries. The easiest way of -doing this is by using the [influxdb-management](https://gitlab.com/gitlab-org/influxdb-management) -repository. - -To use this repository you must first clone it: - -``` -git clone https://gitlab.com/gitlab-org/influxdb-management.git -cd influxdb-management -``` - -Next you must install the required dependencies: - -``` -gem install bundler -bundle install -``` - -Now you must configure the repository by first copying `.env.example` to `.env` -and then editing the `.env` file to contain the correct InfluxDB settings. Once -configured you can simply run `bundle exec rake` and the InfluxDB database will -be configured for you. - -For more information see the [influxdb-management README](https://gitlab.com/gitlab-org/influxdb-management/blob/master/README.md). - -## Import Dashboards - -You can now import a set of default dashboards that will give you a good -start on displaying useful information. GitLab has published a set of default -[Grafana dashboards][grafana-dashboards] to get you started. Clone the -repository or download a zip/tarball, then follow these steps to import each -JSON file. - -Open the dashboard dropdown menu and click 'Import' - - - -Click 'Choose file' and browse to the location where you downloaded or cloned -the dashboard repository. Pick one of the JSON files to import. - - - -Once the dashboard is imported, be sure to click save icon in the top bar. If -you do not save the dashboard after importing it will be removed when you -navigate away. - - - -Repeat this process for each dashboard you wish to import. - -Alternatively you can automatically import all the dashboards into your Grafana -instance. See the README of the [Grafana dashboards][grafana-dashboards] -repository for more information on this process. - -[grafana-dashboards]: https://gitlab.com/gitlab-org/grafana-dashboards - ---- - -Read more on: - -- [Introduction to GitLab Performance Monitoring](introduction.md) -- [GitLab Configuration](gitlab_configuration.md) -- [InfluxDB Installation/Configuration](influxdb_configuration.md) -- [InfluxDB Schema](influxdb_schema.md) +This document was moved to [administration/monitoring/performance/grafana_configuration](../../administration/monitoring/performance/grafana_configuration.md). diff --git a/doc/monitoring/performance/influxdb_configuration.md b/doc/monitoring/performance/influxdb_configuration.md index c30cd2950d8..15fd275e916 100644 --- a/doc/monitoring/performance/influxdb_configuration.md +++ b/doc/monitoring/performance/influxdb_configuration.md @@ -1,193 +1 @@ -# InfluxDB Configuration - -The default settings provided by [InfluxDB] are not sufficient for a high traffic -GitLab environment. The settings discussed in this document are based on the -settings GitLab uses for GitLab.com, depending on your own needs you may need to -further adjust them. - -If you are intending to run InfluxDB on the same server as GitLab, make sure -you have plenty of RAM since InfluxDB can use quite a bit depending on traffic. - -Unless you are going with a budget setup, it's advised to run it separately. - -## Requirements - -- InfluxDB 0.9.5 or newer -- A fairly modern version of Linux -- At least 4GB of RAM -- At least 10GB of storage for InfluxDB data - -Note that the RAM and storage requirements can differ greatly depending on the -amount of data received/stored. To limit the amount of stored data users can -look into [InfluxDB Retention Policies][influxdb-retention]. - -## Installation - -Installing InfluxDB is out of the scope of this document. Please refer to the -[InfluxDB documentation]. - -## InfluxDB Server Settings - -Since InfluxDB has many settings that users may wish to customize themselves -(e.g. what port to run InfluxDB on), we'll only cover the essentials. - -The configuration file in question is usually located at -`/etc/influxdb/influxdb.conf`. Whenever you make a change in this file, -InfluxDB needs to be restarted. - -### Storage Engine - -InfluxDB comes with different storage engines and as of InfluxDB 0.9.5 a new -storage engine is available, called [TSM Tree]. All users **must** use the new -`tsm1` storage engine as this [will be the default engine][tsm1-commit] in -upcoming InfluxDB releases. - -Make sure you have the following in your configuration file: - -``` -[data] - dir = "/var/lib/influxdb/data" - engine = "tsm1" -``` - -### Admin Panel - -Production environments should have the InfluxDB admin panel **disabled**. This -feature can be disabled by adding the following to your InfluxDB configuration -file: - -``` -[admin] - enabled = false -``` - -### HTTP - -HTTP is required when using the [InfluxDB CLI] or other tools such as Grafana, -thus it should be enabled. When enabling make sure to _also_ enable -authentication: - -``` -[http] - enabled = true - auth-enabled = true -``` - -_**Note:** Before you enable authentication, you might want to [create an -admin user](#create-a-new-admin-user)._ - -### UDP - -GitLab writes data to InfluxDB via UDP and thus this must be enabled. Enabling -UDP can be done using the following settings: - -``` -[[udp]] - enabled = true - bind-address = ":8089" - database = "gitlab" - batch-size = 1000 - batch-pending = 5 - batch-timeout = "1s" - read-buffer = 209715200 -``` - -This does the following: - -1. Enable UDP and bind it to port 8089 for all addresses. -2. Store any data received in the "gitlab" database. -3. Define a batch of points to be 1000 points in size and allow a maximum of - 5 batches _or_ flush them automatically after 1 second. -4. Define a UDP read buffer size of 200 MB. - -One of the most important settings here is the UDP read buffer size as if this -value is set too low, packets will be dropped. You must also make sure the OS -buffer size is set to the same value, the default value is almost never enough. - -To set the OS buffer size to 200 MB, on Linux you can run the following command: - -```bash -sysctl -w net.core.rmem_max=209715200 -``` - -To make this permanent, add the following to `/etc/sysctl.conf` and restart the -server: - -```bash -net.core.rmem_max=209715200 -``` - -It is **very important** to make sure the buffer sizes are large enough to -handle all data sent to InfluxDB as otherwise you _will_ lose data. The above -buffer sizes are based on the traffic for GitLab.com. Depending on the amount of -traffic, users may be able to use a smaller buffer size, but we highly recommend -using _at least_ 100 MB. - -When enabling UDP, users should take care to not expose the port to the public, -as doing so will allow anybody to write data into your InfluxDB database (as -[InfluxDB's UDP protocol][udp] doesn't support authentication). We recommend either -whitelisting the allowed IP addresses/ranges, or setting up a VLAN and only -allowing traffic from members of said VLAN. - -## Create a new admin user - -If you want to [enable authentication](#http), you might want to [create an -admin user][influx-admin]: - -``` -influx -execute "CREATE USER jeff WITH PASSWORD '1234' WITH ALL PRIVILEGES" -``` - -## Create the `gitlab` database - -Once you get InfluxDB up and running, you need to create a database for GitLab. -Make sure you have changed the [storage engine](#storage-engine) to `tsm1` -before creating a database. - -_**Note:** If you [created an admin user](#create-a-new-admin-user) and enabled -[HTTP authentication](#http), remember to append the username (`-username <username>`) -and password (`-password <password>`) you set earlier to the commands below._ - -Run the following command to create a database named `gitlab`: - -```bash -influx -execute 'CREATE DATABASE gitlab' -``` - -The name **must** be `gitlab`, do not use any other name. - -Next, make sure that the database was successfully created: - -```bash -influx -execute 'SHOW DATABASES' -``` - -The output should be similar to: - -``` -name: databases ---------------- -name -_internal -gitlab -``` - -That's it! Now your GitLab instance should send data to InfluxDB. - ---- - -Read more on: - -- [Introduction to GitLab Performance Monitoring](introduction.md) -- [GitLab Configuration](gitlab_configuration.md) -- [InfluxDB Schema](influxdb_schema.md) -- [Grafana Install/Configuration](grafana_configuration.md) - -[influxdb-retention]: https://docs.influxdata.com/influxdb/v0.9/query_language/database_management/#retention-policy-management -[influxdb documentation]: https://docs.influxdata.com/influxdb/v0.9/ -[influxdb cli]: https://docs.influxdata.com/influxdb/v0.9/tools/shell/ -[udp]: https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/ -[influxdb]: https://influxdata.com/time-series-platform/influxdb/ -[tsm tree]: https://influxdata.com/blog/new-storage-engine-time-structured-merge-tree/ -[tsm1-commit]: https://github.com/influxdata/influxdb/commit/15d723dc77651bac83e09e2b1c94be480966cb0d -[influx-admin]: https://docs.influxdata.com/influxdb/v0.9/administration/authentication_and_authorization/#create-a-new-admin-user +This document was moved to [administration/monitoring/performance/influxdb_configuration](../../administration/monitoring/performance/influxdb_configuration.md). diff --git a/doc/monitoring/performance/influxdb_schema.md b/doc/monitoring/performance/influxdb_schema.md index eff0e29f58d..e53f9701dc3 100644 --- a/doc/monitoring/performance/influxdb_schema.md +++ b/doc/monitoring/performance/influxdb_schema.md @@ -1,97 +1 @@ -# InfluxDB Schema - -The following measurements are currently stored in InfluxDB: - -- `PROCESS_file_descriptors` -- `PROCESS_gc_statistics` -- `PROCESS_memory_usage` -- `PROCESS_method_calls` -- `PROCESS_object_counts` -- `PROCESS_transactions` -- `PROCESS_views` -- `events` - -Here, `PROCESS` is replaced with either `rails` or `sidekiq` depending on the -process type. In all series, any form of duration is stored in milliseconds. - -## PROCESS_file_descriptors - -This measurement contains the number of open file descriptors over time. The -value field `value` contains the number of descriptors. - -## PROCESS_gc_statistics - -This measurement contains Ruby garbage collection statistics such as the amount -of minor/major GC runs (relative to the last sampling interval), the time spent -in garbage collection cycles, and all fields/values returned by `GC.stat`. - -## PROCESS_memory_usage - -This measurement contains the process' memory usage (in bytes) over time. The -value field `value` contains the number of bytes. - -## PROCESS_method_calls - -This measurement contains the methods called during a transaction along with -their duration, and a name of the transaction action that invoked the method (if -available). The method call duration is stored in the value field `duration`, -while the method name is stored in the tag `method`. The tag `action` contains -the full name of the transaction action. Both the `method` and `action` fields -are in the following format: - -``` -ClassName#method_name -``` - -For example, a method called by the `show` method in the `UsersController` class -would have `action` set to `UsersController#show`. - -## PROCESS_object_counts - -This measurement is used to store retained Ruby objects (per class) and the -amount of retained objects. The number of objects is stored in the `count` value -field while the class name is stored in the `type` tag. - -## PROCESS_transactions - -This measurement is used to store basic transaction details such as the time it -took to complete a transaction, how much time was spent in SQL queries, etc. The -following value fields are available: - -| Value | Description | -| ----- | ----------- | -| `duration` | The total duration of the transaction | -| `allocated_memory` | The amount of bytes allocated while the transaction was running. This value is only reliable when using single-threaded application servers | -| `method_duration` | The total time spent in method calls | -| `sql_duration` | The total time spent in SQL queries | -| `view_duration` | The total time spent in views | - -## PROCESS_views - -This measurement is used to store view rendering timings for a transaction. The -following value fields are available: - -| Value | Description | -| ----- | ----------- | -| `duration` | The rendering time of the view | -| `view` | The path of the view, relative to the application's root directory | - -The `action` tag contains the action name of the transaction that rendered the -view. - -## events - -This measurement is used to store generic events such as the number of Git -pushes, Emails sent, etc. Each point in this measurement has a single value -field called `count`. The value of this field is simply set to `1`. Each point -also has at least one tag: `event`. This tag's value is set to the event name. -Depending on the event type additional tags may be available as well. - ---- - -Read more on: - -- [Introduction to GitLab Performance Monitoring](introduction.md) -- [GitLab Configuration](gitlab_configuration.md) -- [InfluxDB Configuration](influxdb_configuration.md) -- [Grafana Install/Configuration](grafana_configuration.md) +This document was moved to [administration/monitoring/performance/influxdb_schema](../../administration/monitoring/performance/influxdb_schema.md). diff --git a/doc/monitoring/performance/introduction.md b/doc/monitoring/performance/introduction.md index 79904916b7e..ae88baa0c14 100644 --- a/doc/monitoring/performance/introduction.md +++ b/doc/monitoring/performance/introduction.md @@ -1,65 +1 @@ -# GitLab Performance Monitoring - -GitLab comes with its own application performance measuring system as of GitLab -8.4, simply called "GitLab Performance Monitoring". GitLab Performance Monitoring is available in both the -Community and Enterprise editions. - -Apart from this introduction, you are advised to read through the following -documents in order to understand and properly configure GitLab Performance Monitoring: - -- [GitLab Configuration](gitlab_configuration.md) -- [InfluxDB Install/Configuration](influxdb_configuration.md) -- [InfluxDB Schema](influxdb_schema.md) -- [Grafana Install/Configuration](grafana_configuration.md) - -## Introduction to GitLab Performance Monitoring - -GitLab Performance Monitoring makes it possible to measure a wide variety of statistics -including (but not limited to): - -- The time it took to complete a transaction (a web request or Sidekiq job). -- The time spent in running SQL queries and rendering HAML views. -- The time spent executing (instrumented) Ruby methods. -- Ruby object allocations, and retained objects in particular. -- System statistics such as the process' memory usage and open file descriptors. -- Ruby garbage collection statistics. - -Metrics data is written to [InfluxDB][influxdb] over [UDP][influxdb-udp]. Stored -data can be visualized using [Grafana][grafana] or any other application that -supports reading data from InfluxDB. Alternatively data can be queried using the -InfluxDB CLI. - -## Metric Types - -Two types of metrics are collected: - -1. Transaction specific metrics. -1. Sampled metrics, collected at a certain interval in a separate thread. - -### Transaction Metrics - -Transaction metrics are metrics that can be associated with a single -transaction. This includes statistics such as the transaction duration, timings -of any executed SQL queries, time spent rendering HAML views, etc. These metrics -are collected for every Rack request and Sidekiq job processed. - -### Sampled Metrics - -Sampled metrics are metrics that can't be associated with a single transaction. -Examples include garbage collection statistics and retained Ruby objects. These -metrics are collected at a regular interval. This interval is made up out of two -parts: - -1. A user defined interval. -1. A randomly generated offset added on top of the interval, the same offset - can't be used twice in a row. - -The actual interval can be anywhere between a half of the defined interval and a -half above the interval. For example, for a user defined interval of 15 seconds -the actual interval can be anywhere between 7.5 and 22.5. The interval is -re-generated for every sampling run instead of being generated once and re-used -for the duration of the process' lifetime. - -[influxdb]: https://influxdata.com/time-series-platform/influxdb/ -[influxdb-udp]: https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/ -[grafana]: http://grafana.org/ +This document was moved to [administration/monitoring/performance/introduction](../../administration/monitoring/performance/introduction.md). diff --git a/doc/operations/README.md b/doc/operations/README.md index 6a35dab7b6c..58f16aff7bd 100644 --- a/doc/operations/README.md +++ b/doc/operations/README.md @@ -1,5 +1 @@ -# GitLab operations - -- [Sidekiq MemoryKiller](sidekiq_memory_killer.md) -- [Cleaning up Redis sessions](cleaning_up_redis_sessions.md) -- [Understanding Unicorn and unicorn-worker-killer](unicorn.md) +This document was moved to [administration/operations](../administration/operations.md). diff --git a/doc/operations/cleaning_up_redis_sessions.md b/doc/operations/cleaning_up_redis_sessions.md index 93521e976d5..2a1d0a8c8eb 100644 --- a/doc/operations/cleaning_up_redis_sessions.md +++ b/doc/operations/cleaning_up_redis_sessions.md @@ -1,52 +1 @@ -# Cleaning up stale Redis sessions - -Since version 6.2, GitLab stores web user sessions as key-value pairs in Redis. -Prior to GitLab 7.3, user sessions did not automatically expire from Redis. If -you have been running a large GitLab server (thousands of users) since before -GitLab 7.3 we recommend cleaning up stale sessions to compact the Redis -database after you upgrade to GitLab 7.3. You can also perform a cleanup while -still running GitLab 7.2 or older, but in that case new stale sessions will -start building up again after you clean up. - -In GitLab versions prior to 7.3.0, the session keys in Redis are 16-byte -hexadecimal values such as '976aa289e2189b17d7ef525a6702ace9'. Starting with -GitLab 7.3.0, the keys are -prefixed with 'session:gitlab:', so they would look like -'session:gitlab:976aa289e2189b17d7ef525a6702ace9'. Below we describe how to -remove the keys in the old format. - -First we define a shell function with the proper Redis connection details. - -``` -rcli() { - # This example works for Omnibus installations of GitLab 7.3 or newer. For an - # installation from source you will have to change the socket path and the - # path to redis-cli. - sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket "$@" -} - -# test the new shell function; the response should be PONG -rcli ping -``` - -Now we do a search to see if there are any session keys in the old format for -us to clean up. - -``` -# returns the number of old-format session keys in Redis -rcli keys '*' | grep '^[a-f0-9]\{32\}$' | wc -l -``` - -If the number is larger than zero, you can proceed to expire the keys from -Redis. If the number is zero there is nothing to clean up. - -``` -# Tell Redis to expire each matched key after 600 seconds. -rcli keys '*' | grep '^[a-f0-9]\{32\}$' | awk '{ print "expire", $0, 600 }' | rcli -# This will print '(integer) 1' for each key that gets expired. -``` - -Over the next 15 minutes (10 minutes expiry time plus 5 minutes Redis -background save interval) your Redis database will be compacted. If you are -still using GitLab 7.2, users who are not clicking around in GitLab during the -10 minute expiry window will be signed out of GitLab. +This document was moved to [administration/operations/cleaning_up_redis_sessions](../administration/operations/cleaning_up_redis_sessions.md). diff --git a/doc/operations/moving_repositories.md b/doc/operations/moving_repositories.md index 54adb99386a..c54bca324a5 100644 --- a/doc/operations/moving_repositories.md +++ b/doc/operations/moving_repositories.md @@ -1,180 +1 @@ -# Moving repositories managed by GitLab - -Sometimes you need to move all repositories managed by GitLab to -another filesystem or another server. In this document we will look -at some of the ways you can copy all your repositories from -`/var/opt/gitlab/git-data/repositories` to `/mnt/gitlab/repositories`. - -We will look at three scenarios: the target directory is empty, the -target directory contains an outdated copy of the repositories, and -how to deal with thousands of repositories. - -**Each of the approaches we list can/will overwrite data in the -target directory `/mnt/gitlab/repositories`. Do not mix up the -source and the target.** - -## Target directory is empty: use a tar pipe - -If the target directory `/mnt/gitlab/repositories` is empty the -simplest thing to do is to use a tar pipe. This method has low -overhead and tar is almost always already installed on your system. -However, it is not possible to resume an interrupted tar pipe: if -that happens then all data must be copied again. - -``` -# As the git user -tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\ - tar -C /mnt/gitlab/repositories -xf - -``` - -If you want to see progress, replace `-xf` with `-xvf`. - -### Tar pipe to another server - -You can also use a tar pipe to copy data to another server. If your -'git' user has SSH access to the newserver as 'git@newserver', you -can pipe the data through SSH. - -``` -# As the git user -tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\ - ssh git@newserver tar -C /mnt/gitlab/repositories -xf - -``` - -If you want to compress the data before it goes over the network -(which will cost you CPU cycles) you can replace `ssh` with `ssh -C`. - -## The target directory contains an outdated copy of the repositories: use rsync - -If the target directory already contains a partial / outdated copy -of the repositories it may be wasteful to copy all the data again -with tar. In this scenario it is better to use rsync. This utility -is either already installed on your system or easily installable -via apt, yum etc. - -``` -# As the 'git' user -rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ - /mnt/gitlab/repositories -``` - -The `/.` in the command above is very important, without it you can -easily get the wrong directory structure in the target directory. -If you want to see progress, replace `-a` with `-av`. - -### Single rsync to another server - -If the 'git' user on your source system has SSH access to the target -server you can send the repositories over the network with rsync. - -``` -# As the 'git' user -rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ - git@newserver:/mnt/gitlab/repositories -``` - -## Thousands of Git repositories: use one rsync per repository - -Every time you start an rsync job it has to inspect all files in -the source directory, all files in the target directory, and then -decide what files to copy or not. If the source or target directory -has many contents this startup phase of rsync can become a burden -for your GitLab server. In cases like this you can make rsync's -life easier by dividing its work in smaller pieces, and sync one -repository at a time. - -In addition to rsync we will use [GNU -Parallel](http://www.gnu.org/software/parallel/). This utility is -not included in GitLab so you need to install it yourself with apt -or yum. Also note that the GitLab scripts we used below were added -in GitLab 8.1. - -** This process does not clean up repositories at the target location that no -longer exist at the source. ** If you start using your GitLab instance with -`/mnt/gitlab/repositories`, you need to run `gitlab-rake gitlab:cleanup:repos` -after switching to the new repository storage directory. - -### Parallel rsync for all repositories known to GitLab - -This will sync repositories with 10 rsync processes at a time. We keep -track of progress so that the transfer can be restarted if necessary. - -First we create a new directory, owned by 'git', to hold transfer -logs. We assume the directory is empty before we start the transfer -procedure, and that we are the only ones writing files in it. - -``` -# Omnibus -sudo mkdir /var/opt/gitlab/transfer-logs -sudo chown git:git /var/opt/gitlab/transfer-logs - -# Source -sudo -u git -H mkdir /home/git/transfer-logs -``` - -We seed the process with a list of the directories we want to copy. - -``` -# Omnibus -sudo -u git sh -c 'gitlab-rake gitlab:list_repos > /var/opt/gitlab/transfer-logs/all-repos-$(date +%s).txt' - -# Source -cd /home/git/gitlab -sudo -u git -H sh -c 'bundle exec rake gitlab:list_repos > /home/git/transfer-logs/all-repos-$(date +%s).txt' -``` - -Now we can start the transfer. The command below is idempotent, and -the number of jobs done by GNU Parallel should converge to zero. If it -does not some repositories listed in all-repos-1234.txt may have been -deleted/renamed before they could be copied. - -``` -# Omnibus -sudo -u git sh -c ' -cat /var/opt/gitlab/transfer-logs/* | sort | uniq -u |\ - /usr/bin/env JOBS=10 \ - /opt/gitlab/embedded/service/gitlab-rails/bin/parallel-rsync-repos \ - /var/opt/gitlab/transfer-logs/success-$(date +%s).log \ - /var/opt/gitlab/git-data/repositories \ - /mnt/gitlab/repositories -' - -# Source -cd /home/git/gitlab -sudo -u git -H sh -c ' -cat /home/git/transfer-logs/* | sort | uniq -u |\ - /usr/bin/env JOBS=10 \ - bin/parallel-rsync-repos \ - /home/git/transfer-logs/success-$(date +%s).log \ - /home/git/repositories \ - /mnt/gitlab/repositories -` -``` - -### Parallel rsync only for repositories with recent activity - -Suppose you have already done one sync that started after 2015-10-1 12:00 UTC. -Then you might only want to sync repositories that were changed via GitLab -_after_ that time. You can use the 'SINCE' variable to tell 'rake -gitlab:list_repos' to only print repositories with recent activity. - -``` -# Omnibus -sudo gitlab-rake gitlab:list_repos SINCE='2015-10-1 12:00 UTC' |\ - sudo -u git \ - /usr/bin/env JOBS=10 \ - /opt/gitlab/embedded/service/gitlab-rails/bin/parallel-rsync-repos \ - success-$(date +%s).log \ - /var/opt/gitlab/git-data/repositories \ - /mnt/gitlab/repositories - -# Source -cd /home/git/gitlab -sudo -u git -H bundle exec rake gitlab:list_repos SINCE='2015-10-1 12:00 UTC' |\ - sudo -u git -H \ - /usr/bin/env JOBS=10 \ - bin/parallel-rsync-repos \ - success-$(date +%s).log \ - /home/git/repositories \ - /mnt/gitlab/repositories -``` +This document was moved to [administration/operations/moving_repositories](../administration/operations/moving_repositories.md). diff --git a/doc/operations/sidekiq_memory_killer.md b/doc/operations/sidekiq_memory_killer.md index b5e78348989..cf7c3b2e2ed 100644 --- a/doc/operations/sidekiq_memory_killer.md +++ b/doc/operations/sidekiq_memory_killer.md @@ -1,40 +1 @@ -# Sidekiq MemoryKiller - -The GitLab Rails application code suffers from memory leaks. For web requests -this problem is made manageable using -[unicorn-worker-killer](https://github.com/kzk/unicorn-worker-killer) which -restarts Unicorn worker processes in between requests when needed. The Sidekiq -MemoryKiller applies the same approach to the Sidekiq processes used by GitLab -to process background jobs. - -Unlike unicorn-worker-killer, which is enabled by default for all GitLab -installations since GitLab 6.4, the Sidekiq MemoryKiller is enabled by default -_only_ for Omnibus packages. The reason for this is that the MemoryKiller -relies on Runit to restart Sidekiq after a memory-induced shutdown and GitLab -installations from source do not all use Runit or an equivalent. - -With the default settings, the MemoryKiller will cause a Sidekiq restart no -more often than once every 15 minutes, with the restart causing about one -minute of delay for incoming background jobs. - -## Configuring the MemoryKiller - -The MemoryKiller is controlled using environment variables. - -- `SIDEKIQ_MEMORY_KILLER_MAX_RSS`: if this variable is set, and its value is - greater than 0, then after each Sidekiq job, the MemoryKiller will check the - RSS of the Sidekiq process that executed the job. If the RSS of the Sidekiq - process (expressed in kilobytes) exceeds SIDEKIQ_MEMORY_KILLER_MAX_RSS, a - delayed shutdown is triggered. The default value for Omnibus packages is set - [in the omnibus-gitlab - repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb). -- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`: defaults 900 seconds (15 minutes). When - a shutdown is triggered, the Sidekiq process will keep working normally for - another 15 minutes. -- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. When the grace - time has expired, the MemoryKiller tells Sidekiq to stop accepting new jobs. - Existing jobs get 30 seconds to finish. After that, the MemoryKiller tells - Sidekiq to shut down, and an external supervision mechanism (e.g. Runit) must - restart Sidekiq. -- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_SIGNAL`: defaults to `SIGKILL`. The name of - the final signal sent to the Sidekiq process when we want it to shut down. +This document was moved to [administration/operations/sidekiq_memory_killer](../administration/operations/sidekiq_memory_killer.md). diff --git a/doc/operations/unicorn.md b/doc/operations/unicorn.md index bad61151bda..fbc9697b755 100644 --- a/doc/operations/unicorn.md +++ b/doc/operations/unicorn.md @@ -1,86 +1 @@ -# Understanding Unicorn and unicorn-worker-killer - -## Unicorn - -GitLab uses [Unicorn](http://unicorn.bogomips.org/), a pre-forking Ruby web -server, to handle web requests (web browsers and Git HTTP clients). Unicorn is -a daemon written in Ruby and C that can load and run a Ruby on Rails -application; in our case the Rails application is GitLab Community Edition or -GitLab Enterprise Edition. - -Unicorn has a multi-process architecture to make better use of available CPU -cores (processes can run on different cores) and to have stronger fault -tolerance (most failures stay isolated in only one process and cannot take down -GitLab entirely). On startup, the Unicorn 'master' process loads a clean Ruby -environment with the GitLab application code, and then spawns 'workers' which -inherit this clean initial environment. The 'master' never handles any -requests, that is left to the workers. The operating system network stack -queues incoming requests and distributes them among the workers. - -In a perfect world, the master would spawn its pool of workers once, and then -the workers handle incoming web requests one after another until the end of -time. In reality, worker processes can crash or time out: if the master notices -that a worker takes too long to handle a request it will terminate the worker -process with SIGKILL ('kill -9'). No matter how the worker process ended, the -master process will replace it with a new 'clean' process again. Unicorn is -designed to be able to replace 'crashed' workers without dropping user -requests. - -This is what a Unicorn worker timeout looks like in `unicorn_stderr.log`. The -master process has PID 56227 below. - -``` -[2015-06-05T10:58:08.660325 #56227] ERROR -- : worker=10 PID:53009 timeout (61s > 60s), killing -[2015-06-05T10:58:08.699360 #56227] ERROR -- : reaped #<Process::Status: pid 53009 SIGKILL (signal 9)> worker=10 -[2015-06-05T10:58:08.708141 #62538] INFO -- : worker=10 spawned pid=62538 -[2015-06-05T10:58:08.708824 #62538] INFO -- : worker=10 ready -``` - -### Tunables - -The main tunables for Unicorn are the number of worker processes and the -request timeout after which the Unicorn master terminates a worker process. -See the [omnibus-gitlab Unicorn settings -documentation](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/unicorn.md) -if you want to adjust these settings. - -## unicorn-worker-killer - -GitLab has memory leaks. These memory leaks manifest themselves in long-running -processes, such as Unicorn workers. (The Unicorn master process is not known to -leak memory, probably because it does not handle user requests.) - -To make these memory leaks manageable, GitLab comes with the -[unicorn-worker-killer gem](https://github.com/kzk/unicorn-worker-killer). This -gem [monkey-patches](https://en.wikipedia.org/wiki/Monkey_patch) the Unicorn -workers to do a memory self-check after every 16 requests. If the memory of the -Unicorn worker exceeds a pre-set limit then the worker process exits. The -Unicorn master then automatically replaces the worker process. - -This is a robust way to handle memory leaks: Unicorn is designed to handle -workers that 'crash' so no user requests will be dropped. The -unicorn-worker-killer gem is designed to only terminate a worker process _in -between requests_, so no user requests are affected. - -This is what a Unicorn worker memory restart looks like in unicorn_stderr.log. -You see that worker 4 (PID 125918) is inspecting itself and decides to exit. -The threshold memory value was 254802235 bytes, about 250MB. With GitLab this -threshold is a random value between 200 and 250 MB. The master process (PID -117565) then reaps the worker process and spawns a new 'worker 4' with PID -127549. - -``` -[2015-06-05T12:07:41.828374 #125918] WARN -- : #<Unicorn::HttpServer:0x00000002734770>: worker (pid: 125918) exceeds memory limit (256413696 bytes > 254802235 bytes) -[2015-06-05T12:07:41.828472 #125918] WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 125918) alive: 23 sec (trial 1) -[2015-06-05T12:07:42.025916 #117565] INFO -- : reaped #<Process::Status: pid 125918 exit 0> worker=4 -[2015-06-05T12:07:42.034527 #127549] INFO -- : worker=4 spawned pid=127549 -[2015-06-05T12:07:42.035217 #127549] INFO -- : worker=4 ready -``` - -One other thing that stands out in the log snippet above, taken from -GitLab.com, is that 'worker 4' was serving requests for only 23 seconds. This -is a normal value for our current GitLab.com setup and traffic. - -The high frequency of Unicorn memory restarts on some GitLab sites can be a -source of confusion for administrators. Usually they are a [red -herring](https://en.wikipedia.org/wiki/Red_herring). +This document was moved to [administration/operations/unicorn](../administration/operations/unicorn.md). diff --git a/doc/project_services/img/builds_emails_service.png b/doc/project_services/img/builds_emails_service.png Binary files differindex 88943dc410e..440728795be 100644 --- a/doc/project_services/img/builds_emails_service.png +++ b/doc/project_services/img/builds_emails_service.png diff --git a/doc/raketasks/backup_hrz.png b/doc/raketasks/backup_hrz.png Binary files differindex 42084717ebe..287587609a1 100644 --- a/doc/raketasks/backup_hrz.png +++ b/doc/raketasks/backup_hrz.png diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index 3f4056dc440..fc0cd1b8af2 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -2,34 +2,51 @@  -## Create a backup of the GitLab system - -A backup creates an archive file that contains the database, all repositories and all attachments. -This archive will be saved in backup_path (see `config/gitlab.yml`). -The filename will be `[TIMESTAMP]_gitlab_backup.tar`. This timestamp can be used to restore an specific backup. -You can only restore a backup to exactly the same version of GitLab that you created it -on, for example 7.2.1. The best way to migrate your repositories from one server to +An application data backup creates an archive file that contains the database, +all repositories and all attachments. +This archive will be saved in `backup_path`, which is specified in the +`config/gitlab.yml` file. +The filename will be `[TIMESTAMP]_gitlab_backup.tar`, where `TIMESTAMP` +identifies the time at which each backup was created. + +You can only restore a backup to exactly the same version of GitLab on which it +was created. The best way to migrate your repositories from one server to another is through backup restore. -You need to keep separate copies of `/etc/gitlab/gitlab-secrets.json` and -`/etc/gitlab/gitlab.rb` (for omnibus packages) or -`/home/git/gitlab/config/secrets.yml` (for installations from source). This file -contains the database encryption keys used for two-factor authentication and CI -secret variables, among other things. If you restore a GitLab backup without -restoring the database encryption key, users who have two-factor authentication -enabled will lose access to your GitLab server. +To restore a backup, you will also need to restore `/etc/gitlab/gitlab-secrets.json` +(for omnibus packages) or `/home/git/gitlab/.secret` (for installations +from source). This file contains the database encryption key and CI secret +variables used for two-factor authentication. If you fail to restore this +encryption key file along with the application data backup, users with two-factor +authentication enabled will lose access to your GitLab server. + +## Create a backup of the GitLab system +Use this command if you've installed GitLab with the Omnibus package: ``` -# use this command if you've installed GitLab with the Omnibus package sudo gitlab-rake gitlab:backup:create - -# if you've installed GitLab from source +``` +Use this if you've installed GitLab from source: +``` sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production ``` +If you are running GitLab within a Docker container, you can run the backup from the host: +``` +docker -t exec <container name> gitlab-rake gitlab:backup:create +``` + +You can specify that portions of the application data be skipped using the +environment variable `SKIP`. You can skip: + +- `db` (database) +- `uploads` (attachments) +- `repositories` (Git repositories data) +- `builds` (CI build output logs) +- `artifacts` (CI build artifacts) +- `lfs` (LFS objects) +- `registry` (Container Registry images) -Also you can choose what should be backed up by adding environment variable SKIP. Available options: db, -uploads (attachments), repositories, builds(CI build output logs), artifacts (CI build artifacts), lfs (LFS objects). -Use a comma to specify several options at the same time. +Separate multiple data types to skip using a comma. For example: ``` sudo gitlab-rake gitlab:backup:create SKIP=db,uploads @@ -69,7 +86,7 @@ Deleting old backups... [SKIPPING] Starting with GitLab 7.4 you can let the backup script upload the '.tar' file it creates. It uses the [Fog library](http://fog.io/) to perform the upload. In the example below we use Amazon S3 for storage. -But Fog also lets you use [other storage providers](http://fog.io/storage/). +Fog also supports [other storage providers](http://fog.io/storage/). For omnibus packages: @@ -161,7 +178,7 @@ with the name of your bucket: ### Uploading to locally mounted shares You may also send backups to a mounted share (`NFS` / `CIFS` / `SMB` / etc.) by -using the [`Local`](https://github.com/fog/fog-local#usage) storage provider. +using the Fog [`Local`](https://github.com/fog/fog-local#usage) storage provider. The directory pointed to by the `local_root` key **must** be owned by the `git` user **when mounted** (mounting with the `uid=` of the `git` user for `CIFS` and `SMB`) or the user that you are executing the backup tasks under (for omnibus @@ -228,7 +245,7 @@ of using encryption in the first place! If you use an Omnibus package please see the [instructions in the readme to backup your configuration](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md#backup-and-restore-omnibus-gitlab-configuration). If you have a cookbook installation there should be a copy of your configuration in Chef. -If you have an installation from source, please consider backing up your `config/secrets.yml` file, `gitlab.yml` file, any SSL keys and certificates, and your [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). +If you installed from source, please consider backing up your `config/secrets.yml` file, `gitlab.yml` file, any SSL keys and certificates, and your [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). At the very **minimum** you should backup `/etc/gitlab/gitlab.rb` and `/etc/gitlab/gitlab-secrets.json` (Omnibus), or diff --git a/doc/raketasks/user_management.md b/doc/raketasks/user_management.md index 8a5e2d6e16b..044b104f5c2 100644 --- a/doc/raketasks/user_management.md +++ b/doc/raketasks/user_management.md @@ -70,3 +70,18 @@ sudo gitlab-rake gitlab:two_factor:disable_for_all_users # installation from source bundle exec rake gitlab:two_factor:disable_for_all_users RAILS_ENV=production ``` + +## Clear authentication tokens for all users. Important! Data loss! + +Clear authentication tokens for all users in the GitLab database. This +task is useful if your users' authentication tokens might have been exposed in +any way. All the existing tokens will become invalid, and new tokens are +automatically generated upon sign-in or user modification. + +``` +# omnibus-gitlab +sudo gitlab-rake gitlab:users:clear_all_authentication_tokens + +# installation from source +bundle exec rake gitlab:users:clear_all_authentication_tokens RAILS_ENV=production +``` diff --git a/doc/university/README.md b/doc/university/README.md index 6ca1c20c9b2..f5a0dab39fe 100644 --- a/doc/university/README.md +++ b/doc/university/README.md @@ -1,139 +1,216 @@ +# GitLab University -## What is GitLab University +GitLab University is the best place to learn about **Version Control with Git and GitLab**. -_GitLab University_ has as a goal to teach the fundamentals of **Version Control with Git and GitLab** through courses that cover topics which can be mastered in around 2 hours. +It doesn't replace, but accompanies our great [Documentation](http://docs.gitlab.com) +and [Blog Articles](https://about.gitlab.com/blog/). -_University materials don't replace our [Documentation](http://docs.gitlab.com) or [Blog Articles](https://about.gitlab.com/blog/)._ +Would you like to contribute to GitLab University? Then please take a look at our contribution [process](/process) for more information. ---- +## Gitlab University Curriculum + +The curriculum is composed of GitLab videos, screencasts, presentations, projects and external GitLab content hosted on other services and has been organized into the following sections. -### On this page - -+ [GITx] Git -+ [OPSx] DevOps -+ [GLBx] GitLab Basics -+ [INTx] GitLab Integrations -+ [GLFx] GitLab Workflows -+ [GLEx] GitLab Enterprise Edition extra features -+ [GCIx] GitLab CI -+ [ECO] Ecosystem -+ [COM] Competition comparison -+ [SPTx] Support Bootcamp -+ [SLSx] Sales Bootcamp -+ [TRAx] Trainings +1. [GitLab Beginner](#beginner) +1. [GitLab Intermediate](#intermediate) +1. [GitLab Advanced](#advanced) +1. [External Articles](#external) +1. [Resources for GitLab Team Members](#team) --- -+ [GIT1] [Version Control Systems](https://docs.google.com/presentation/d/16sX7hUrCZyOFbpvnrAFrg6tVO5_yT98IgdAqOmXwBho/edit#slide=id.g72f2e4906_2_29) -+ [GIT2] [Operating Systems and How Git Works](https://drive.google.com/a/gitlab.com/file/d/0B41DBToSSIG_OVYxVFJDOGI3Vzg/view?usp=sharing) -+ [GIT3] [Intro to Git](https://www.codeschool.com/account/courses/try-git) +### 1. <a name="beginner"></a> GitLab Beginner ---- +#### 1.1. Version Control and Git -+ [OPS1] [What is Omnibus](https://www.youtube.com/watch?v=XTmpKudd-Oo) -+ [OPS2] [Installing GitLab](https://www.youtube.com/watch?v=Q69YaOjqNhg) -+ [OPS3] [Configuring an external PostgreSQL database](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md#using-a-non-packaged-postgresql-database-management-server) -+ [OPS5] [Importing from Other Tools or SVN](http://doc.gitlab.com/ee/workflow/importing/) -+ [OPS6] [High Availability Documentation](https://about.gitlab.com/high-availability/) -+ [OPS7] [Managing LDAP, Active Directory](https://www.youtube.com/watch?v=HPMjM-14qa8) -+ [OPS8] [Scalability and High Availability](https://www.youtube.com/watch?v=cXRMJJb6sp4&list=PLFGfElNsQthbQu_IWlNOxul0TbS_2JH-e&index=2) -+ [OPS9] [High Availability on AWS](high-availability/aws/README.md) +1. [Version Control Systems](https://docs.google.com/presentation/d/16sX7hUrCZyOFbpvnrAFrg6tVO5_yT98IgdAqOmXwBho/edit#slide=id.g72f2e4906_2_29) +1. [Operating Systems and How Git Works](https://drive.google.com/a/gitlab.com/file/d/0B41DBToSSIG_OVYxVFJDOGI3Vzg/view?usp=sharing) +1. [Code School: An Introduction to Git](https://www.codeschool.com/account/courses/try-git) ---- +#### 1.2. GitLab Basics -+ [GLB1] [Terminology](glossary/README.md) -+ [GLB2] [GitLab Basics](http://doc.gitlab.com/ce/gitlab-basics/README.html) -+ [GLB3] [Demo of GitLab.com](https://www.youtube.com/watch?v=WaiL5DGEMR4) -+ [GLB4] [Create and Add your SSH key to GitLab](https://www.youtube.com/watch?v=54mxyLo3Mqk) -+ [GLB5] [Repositories, Projects and Groups](https://www.youtube.com/watch?v=4TWfh1aKHHw&index=1&list=PLFGfElNsQthbQu_IWlNOxul0TbS_2JH-e) -+ [GLB6] [Creating a Project in GitLab](https://www.youtube.com/watch?v=7p0hrpNaJ14) -+ [GLB7] [Issues and Merge Requests](https://www.youtube.com/watch?v=raXvuwet78M) -+ [GLB8] [Big files in Git (Git LFS, Annex)](https://gitlab.com/gitlab-org/University/blob/master/classes/git_lfs_and_annex.md) +1. [An Overview of GitLab.com - Video](https://www.youtube.com/watch?v=WaiL5DGEMR4) +1. [Why Use Git and GitLab - Slides](https://docs.google.com/a/gitlab.com/presentation/d/1RcZhFmn5VPvoFu6UMxhMOy7lAsToeBZRjLRn0LIdaNc/edit?usp=drive_web) +1. [GitLab Basics - Article](http://doc.gitlab.com/ce/gitlab-basics/README.html) +1. [Git and GitLab Basics - Video](https://www.youtube.com/watch?v=03wb9FvO4Ak&index=5&list=PLFGfElNsQthbQu_IWlNOxul0TbS_2JH-e) +1. [Git and GitLab Basics - Online Course](https://courses.platzi.com/classes/git-gitlab/concepto/part-1/part-23370/material/) +1. [Comparison of GitLab Versions](https://about.gitlab.com/features/#compare) ---- +#### 1.3. Your GitLab Account -+ [INT1] [JIRA and Jenkins integrations in GitLab](https://gitlabmeetings.webex.com/gitlabmeetings/ldr.php?RCID=44b548147a67ab4d8a62274047146415) -+ [INT2] [Integrating JIRA with GitLab](http://doc.gitlab.com/ee/integration/jira.html) -+ [INT3] [Integrating Jenkins with GitLab](http://doc.gitlab.com/ee/integration/jenkins.html) -+ [INT4] [Integrating Bamboo with GitLab](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/project_services/bamboo.md) -+ [INT5] [Documentation on Integrating Slack with GitLab](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/integration/slack.md) +1. [Create a GitLab Account - Online Course](https://courses.platzi.com/classes/git-gitlab/concepto/first-steps/create-an-account-on-gitlab/material/) +1. [Create and Add your SSH key to GitLab - Video](https://www.youtube.com/watch?v=54mxyLo3Mqk) ---- +#### 1.4. GitLab Projects -+ [GLF1] [GitLab Flow](https://www.youtube.com/watch?v=UGotqAUACZA) +1. [Repositories, Projects and Groups - Video](https://www.youtube.com/watch?v=4TWfh1aKHHw&index=1&list=PLFGfElNsQthbQu_IWlNOxul0TbS_2JH-e) +1. [Creating a Project in GitLab - Video](https://www.youtube.com/watch?v=7p0hrpNaJ14) +1. [How to Create Files and Directories](https://about.gitlab.com/2016/02/10/feature-highlight-create-files-and-directories-from-files-page/) +1. [GitLab Todos](https://about.gitlab.com/2016/03/02/gitlab-todos-feature-highlight/) +1. [GitLab's Work in Progress (WIP) Flag](https://about.gitlab.com/2016/01/08/feature-highlight-wip/) ---- +#### 1.5. Migrating from other Source Control -+ [GLE1] [Configuring an external MySQL database](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md#using-a-mysql-database-management-server-enterprise-edition-only) -+ [GLE2] [Managing Permissions within EE](https://www.youtube.com/watch?v=DjUoIrkiNuM) -+ [GLE3] [Upcoming in EE and Big files in Git (Git LFS, Annex)](https://gitlab.com/gitlab-org/University/blob/master/classes/upcoming_in_ee.md) +1. [Migrating from BitBucket/Stash](http://doc.gitlab.com/ee/workflow/importing/import_projects_from_bitbucket.html) +1. [Migrating from GitHub](http://doc.gitlab.com/ee/workflow/importing/import_projects_from_github.html) +1. [Migrating from SVN](http://doc.gitlab.com/ee/workflow/importing/migrating_from_svn.html) +1. [Migrating from Fogbugz](http://doc.gitlab.com/ee/workflow/importing/import_projects_from_fogbugz.html) ---- +#### 1.6. GitLab Inc. -+ [GCI1] [GitLab CI product page](https://about.gitlab.com/gitlab-ci/) -+ [GCI2] [Setting up GitLab Runner For Continuous Integration](https://about.gitlab.com/2016/03/01/gitlab-runner-with-docker/) +1. [About GitLab](https://about.gitlab.com/about/) +1. [GitLab Direction](https://about.gitlab.com/direction/) +1. [GitLab Master Plan](https://about.gitlab.com/2016/09/13/gitlab-master-plan/) +1. [Making GitLab Great for Everyone - Video](https://www.youtube.com/watch?v=GGC40y4vMx0) - Response to "Dear GitHub" letter +1. [Using Innersourcing to Improve Collaboration](https://about.gitlab.com/2014/09/05/innersourcing-using-the-open-source-workflow-to-improve-collaboration-within-an-organization/) +1. [The Software Development Market and GitLab - Video](https://www.youtube.com/watch?v=sXlhgPK1NTY&list=PLFGfElNsQthbQu_IWlNOxul0TbS_2JH-e&index=6) - [Slides](https://docs.google.com/presentation/d/1vCU-NbZWz8NTNK8Vu3y4zGMAHb5DpC8PE5mHtw1PWfI/edit) +1. [The GitLab Book Club](bookclub/index.md) ---- +#### 1.7 Community and Support -+ [COM1] [GitLab compared to other tools](https://about.gitlab.com/comparison/) -+ [COM2] [Compare GitLab versions](https://about.gitlab.com/features/#compare) -+ [COM3] [Innersourcing article](https://about.gitlab.com/2014/09/05/innersourcing-using-the-open-source-workflow-to-improve-collaboration-within-an-organization/) +1. [Getting Help](https://about.gitlab.com/getting-help/) + - Proposing Features and Reporting and Tracking bugs for GitLab + - The GitLab IRC channel, Gitter Chat Room, Community Forum and Mailing List + - Getting Technical Support + - Being part of our Great Community and Contributing to GitLab +1. [Getting Started with the GitLab Development Kit (GDK)](https://about.gitlab.com/2016/06/08/getting-started-with-gitlab-development-kit/) +1. [Contributing Technical Articles to the GitLab Blog](https://about.gitlab.com/2016/01/26/call-for-writers/) +1. [GitLab Training Workshops](https://about.gitlab.com/training) ---- +#### 1.8 GitLab Training Material -+ [ECO1] [Ecosystem Overview](https://www.youtube.com/watch?v=sXlhgPK1NTY&list=PLFGfElNsQthbQu_IWlNOxul0TbS_2JH-e&index=6) -+ [ECO2] [Positioning FAQ](https://about.gitlab.com/handbook/positioning-faq) -+ [ECO3] [GitLab Ecosystem slides](https://docs.google.com/presentation/d/1vCU-NbZWz8NTNK8Vu3y4zGMAHb5DpC8PE5mHtw1PWfI/edit) -+ [ECO4] [Customer Use-Cases](https://about.gitlab.com/handbook/use-cases/) +1. [Git and GitLab Terminology](glossary/README.md) +1. [Git and GitLab Workshop - Slides](https://docs.google.com/presentation/d/1JzTYD8ij9slejV2-TO-NzjCvlvj6mVn9BORePXNJoMI/edit?usp=drive_web) +1. [Git and GitLab Revision](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/university/training/end-user) --- -+ [SPT1] [Support Path](support/README.md) -+ [SPT2] [End User Training Material](https://gitlab.com/gitlab-org/University/blob/master/training/user_training.md) -+ [SPT3] [Materials for Training Sessions](https://gitlab.com/gitlab-org/University/tree/master/training/topics) +### 2. <a name="intermediate"></a> GitLab Intermediate + +#### 2.1 GitLab Pages + +1. [Using any Static Site Generator with GitLab Pages](https://about.gitlab.com/2016/06/17/ssg-overview-gitlab-pages-part-3-examples-ci/) +1. [Securing GitLab Pages with SSL](https://about.gitlab.com/2016/06/24/secure-gitlab-pages-with-startssl/) +1. [GitLab Pages Documentation](http://doc.gitlab.com/ee/pages/README.html) + +#### 2.2. GitLab Issues + +1. [Markdown in GitLab](http://doc.gitlab.com/ce/markdown/markdown.html) +1. [Issues and Merge Requests - Video](https://www.youtube.com/watch?v=raXvuwet78M) +1. [Due Dates and Milestones fro GitLab Issues](https://about.gitlab.com/2016/08/05/feature-highlight-set-dates-for-issues/) +1. [How to Use GitLab Labels](https://about.gitlab.com/2016/08/17/using-gitlab-labels/) +1. [Applying GitLab Labels Automatically](https://about.gitlab.com/2016/08/19/applying-gitlab-labels-automatically/) +1. [GitLab Issue Board - Product Page](https://about.gitlab.com/solutions/issueboard/) +1. [An Overview of GitLab Issue Board](https://about.gitlab.com/2016/08/22/announcing-the-gitlab-issue-board/) +1. [Designing GitLab Issue Board](https://about.gitlab.com/2016/08/31/designing-issue-boards/) +1. [From Idea to Production with GitLab - Video](https://www.youtube.com/watch?v=25pHyknRgEo&index=14&list=PLFGfElNsQthbQu_IWlNOxul0TbS_2JH-e) + +#### 2.3. Continuous Integration + +1. [Operating Systems, Servers, VMs, Containers and Unix - Video](https://www.youtube.com/watch?v=V61kL6IC-zY&index=8&list=PLFGfElNsQthbQu_IWlNOxul0TbS_2JH-e) +1. [GitLab CI - Product Page](https://about.gitlab.com/gitlab-ci/) +1. [Getting started with GitLab and GitLab CI](https://about.gitlab.com/2015/12/14/getting-started-with-gitlab-and-gitlab-ci/) +1. [GitLab Container Registry](https://about.gitlab.com/2016/05/23/gitlab-container-registry/) +1. [GitLab and Docker - Video](https://www.youtube.com/watch?v=ugOrCcbdHko&index=12&list=PLFGfElNsQthbQu_IWlNOxul0TbS_2JH-e) +1. [How we scale GitLab with built in Docker](https://about.gitlab.com/2016/06/21/how-we-scale-gitlab-by-having-docker-built-in/) +1. [Continuous Integration, Delivery, and Deployment with GitLab](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/) +1. [Deployments and Environments](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) +1. [Sequential, Parallel or Custom Pipelines](https://about.gitlab.com/2016/07/29/the-basics-of-gitlab-ci/) +1. [Setting up GitLab Runner For Continuous Integration](https://about.gitlab.com/2016/03/01/gitlab-runner-with-docker/) +1. [Setting up GitLab Runner on DigitalOcean](https://about.gitlab.com/2016/04/19/how-to-set-up-gitlab-runner-on-digitalocean/) +1. [Setting up GitLab CI for iOS projects](https://about.gitlab.com/2016/03/10/setting-up-gitlab-ci-for-ios-projects/) +1. [IBM: Continuous Delivery vs Continuous Deployment - Video](https://www.youtube.com/watch?v=igwFj8PPSnw) +1. [Amazon: Transition to Continuous Delivery - Video](https://www.youtube.com/watch?v=esEFaY0FDKc) +1. See **[Integrations](#integrations)** for integrations with other CI services. + +#### 2.4. Workflow + +1. [GitLab Flow - Video](https://youtu.be/enMumwvLAug?list=PLFGfElNsQthZnwMUFi6rqkyUZkI00OxIV) +1. [GitLab Flow vs Forking in GitLab - Video](https://www.youtube.com/watch?v=UGotqAUACZA) +1. [GitLab Flow Overview](https://about.gitlab.com/2014/09/29/gitlab-flow/) +1. [Always Start with an Issue](https://about.gitlab.com/2016/03/03/start-with-an-issue/) +1. [GitLab Flow Documentation](http://doc.gitlab.com/ee/workflow/gitlab_flow.html) + +#### 2.5. GitLab Comparisons + +1. [GitLab Compared to Other Tools](https://about.gitlab.com/comparison/) +1. [Comparing GitLab Terminology](https://about.gitlab.com/2016/01/27/comparing-terms-gitlab-github-bitbucket/) +1. [GitLab Compared to Atlassian (Recording 2016-03-03) ](https://youtu.be/Nbzp1t45ERo) +1. [GitLab Position FAQ](https://about.gitlab.com/handbook/positioning-faq) +1. [Customer review of GitLab with points on why they prefer GitLab](https://www.enovate.co.uk/web-design-blog/2015/11/25/gitlab-review/) --- -+ [SLS1] [Sales Path (redirect to sales handbook)](https://about.gitlab.com/handbook/sales-onboarding/) -+ [SLS2] [GitLab Direction](https://about.gitlab.com/direction/) +### 3. <a name="advanced"></a> GitLab Advanced + +#### 3.1. Dev Ops + +1. [Xebia Labs: Dev Ops Terminology](https://xebialabs.com/glossary/) +1. [Xebia Labs: Periodic Table of DevOps Tools](https://xebialabs.com/periodic-table-of-devops-tools/) +1. [Puppet Labs: State of Dev Ops 2015 - Book](https://puppetlabs.com/sites/default/files/2015-state-of-devops-report.pdf) + +#### 3.2. Installing GitLab with Omnibus + +1. [What is Omnibus - Video](https://www.youtube.com/watch?v=XTmpKudd-Oo) +1. [How to Install GitLab with Omnibus - Video](https://www.youtube.com/watch?v=Q69YaOjqNhg) +1. [Installing GitLab - Online Course](https://courses.platzi.com/classes/git-gitlab/concepto/part-1/part-3/material/) +1. [Using a Non-Packaged PostgreSQL Database](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md#using-a-non-packaged-postgresql-database-management-server) +1. [Using a MySQL Database](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md#using-a-mysql-database-management-server-enterprise-edition-only) +1. [Installing GitLab on Microsoft Azure](https://about.gitlab.com/2016/07/13/how-to-setup-a-gitlab-instance-on-microsoft-azure/) +1. [Installing GitLab on Digital Ocean](https://about.gitlab.com/2016/04/27/getting-started-with-gitlab-and-digitalocean/) + +#### 3.3. Permissions + +1. [How to Manage Permissions in GitLab EE - Video](https://www.youtube.com/watch?v=DjUoIrkiNuM) + +#### 3.4. Large Files + +1. [Big files in Git (Git LFS, Annex) - Video](https://www.youtube.com/watch?v=DawznUxYDe4) + +#### 3.5. LDAP and Active Directory + +1. [How to Manage LDAP, Active Directory in GitLab - Video](https://www.youtube.com/watch?v=HPMjM-14qa8) + +#### 3.6 Custom Languages + +1. [How to add Syntax Highlighting Support for Custom Langauges to GitLab - Video](how to add support for your favorite language to GitLab) + +#### 3.7. Scalability and High Availability + +1. [Scalability and High Availability - Video](https://www.youtube.com/watch?v=cXRMJJb6sp4&list=PLFGfElNsQthbQu_IWlNOxul0TbS_2JH-e&index=2) +1. [High Availability - Video](https://www.youtube.com/watch?v=36KS808u6bE&index=15&list=PLFGfElNsQthbQu_IWlNOxul0TbS_2JH-e) +1. [High Availability Documentation](https://about.gitlab.com/high-availability/) + +#### 3.8 Cycle Analytics + +1. [GitLab Cycle Analytics Overview](https://about.gitlab.com/2016/09/21/cycle-analytics-feature-highlight/) +1. [GitLab Cycle Analytics - Product Page](https://about.gitlab.com/solutions/cycle-analytics/) + +#### 3.9. <a name="integrations"></a> Integrations + +1. [How to Integrate JIRA and Jenkins with GitLab - Video](https://gitlabmeetings.webex.com/gitlabmeetings/ldr.php?RCID=44b548147a67ab4d8a62274047146415) +1. [How to Integrate Jira with GitLab](http://doc.gitlab.com/ee/integration/jira.html) +1. [How to Integrate Jenkins with GitLab](http://doc.gitlab.com/ee/integration/jenkins.html) +1. [How to Integrate Bamboo with GitLab](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/project_services/bamboo.md) +1. [How to Integrate Slack with GitLab](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/integration/slack.md) +1. [How to Integrate Convox with GitLab](https://about.gitlab.com/2016/06/09/continuous-delivery-with-gitlab-and-convox/) +1. [Getting Started with GitLab and Shippable CI](https://about.gitlab.com/2016/05/05/getting-started-gitlab-and-shippable/) --- -+ [TRA1] [End User Training](training/end-user/README.md) +## 4. <a name="external"></a> External Articles + +1. [2011 WSJ article by Mark Andreeson - Software is Eating the World](http://www.wsj.com/articles/SB10001424053111903480904576512250915629460) +1. [2014 Blog post by Chris Dixon - Software eats software development](http://cdixon.org/2014/04/13/software-eats-software-development/) +1. [2015 Venture Beat article - Actually, Open Source is Eating the World](http://venturebeat.com/2015/12/06/its-actually-open-source-software-thats-eating-the-world/) --- -### External Resources - -+ [DOC] GitLab Documentation - + [Set up and use GitLab Pages](http://doc.gitlab.com/ee/pages/README.html) - + [Markdown Reference](http://doc.gitlab.com/ce/markdown/markdown.html) - -+ [GLW] GitLab Workshop (@ Platzi) - + [GitLab Workshop Part 1: Basics of Git and GitLab](https://courses.platzi.com/classes/git-gitlab/) - + [Create a GitLab Account](https://courses.platzi.com/classes/git-gitlab/concepto/first-steps/create-an-account-on-gitlab/material/) - -+ [GLY] GitLab YouTube Videos - + [Making GitLab Great for Everyone, our response to the Dear GitHub letter](https://www.youtube.com/watch?v=GGC40y4vMx0) - + [Compared to Atlassian (Recorded on 2016-03-03) ](https://youtu.be/Nbzp1t45ERo) - -+ [GLI] GitLab Team-Only Access - + [GitLab architecture for noobs](https://dev.gitlab.org/gitlab/gitlabhq/blob/master/doc/development/architecture.md) - + [Client Assessment of GitLab versus GitHub](https://docs.google.com/a/gitlab.com/spreadsheets/d/18cRF9Y5I6I7Z_ab6qhBEW55YpEMyU4PitZYjomVHM-M/edit?usp=sharing) - -+ [KNT] Slides & Keynotes by GitLabbers & other individuals - + [Why Git and GitLab slide deck](https://docs.google.com/a/gitlab.com/presentation/d/1RcZhFmn5VPvoFu6UMxhMOy7lAsToeBZRjLRn0LIdaNc/) - + [Git Workshop](https://docs.google.com/presentation/d/1JzTYD8ij9slejV2-TO-NzjCvlvj6mVn9BORePXNJoMI/) - -+ Others (not created by GitLab) - + [Dev Ops terminology](https://xebialabs.com/glossary/) - + [Continuous Delivery vs Continuous Deployment](https://www.youtube.com/watch?v=igwFj8PPSnw) - + [Periodic Table of DevOps Tools](https://xebialabs.com/periodic-table-of-devops-tools/) - + [State of Dev Ops 2015 Report by Puppet Labs](https://puppetlabs.com/sites/default/files/2015-state-of-devops-report.pdf) Insightful Chapters to understand the Impact of Continuous Delivery on Performance (Chapter 4), the Application Architecture (Chapter 5) and How IT Managers can help their teams win (Chapter 6). - + [2011 WSJ article by Mark Andreeson - Software is Eating the World](http://www.wsj.com/articles/SB10001424053111903480904576512250915629460) - + [2014 Blog post by Chris Dixon - Software eats software development](http://cdixon.org/2014/04/13/software-eats-software-development/) - + [2015 Venture Beat article - Actually, Open Source is Eating the World](http://venturebeat.com/2015/12/06/its-actually-open-source-software-thats-eating-the-world/) - + [Customer review of GitLab with talking points on why they prefer GitLab](https://www.enovate.co.uk/web-design-blog/2015/11/25/gitlab-review/) - + [3rd party tool comparison](http://technologyconversations.com/2015/10/16/github-vs-gitlabs-vs-bitbucket-server-formerly-stash/) - + [Amazon's transition to Continuous Delivery](https://www.youtube.com/watch?v=esEFaY0FDKc) - + [Article on Continuous Integration from ThoughtWorks](https://www.thoughtworks.com/continuous-integration) +## 5. <a name="team"></a> Resources for GitLab Team Members + +*Some content can only be accessed by GitLab team members* + +1. [Support Path](support/README.md) +1. [Sales Path (redirect to sales handbook)](https://about.gitlab.com/handbook/sales-onboarding/) +1. [GitLab architecture for noobs](https://dev.gitlab.org/gitlab/gitlabhq/blob/master/doc/development/architecture.md) +1. [Client Assessment of GitLab versus GitHub](https://docs.google.com/a/gitlab.com/spreadsheets/d/18cRF9Y5I6I7Z_ab6qhBEW55YpEMyU4PitZYjomVHM-M/edit?usp=sharing) diff --git a/doc/university/bookclub/booklist.md b/doc/university/bookclub/booklist.md new file mode 100644 index 00000000000..c4229832e9f --- /dev/null +++ b/doc/university/bookclub/booklist.md @@ -0,0 +1,113 @@ +# Books + +List of books and resources, that may be worth reading. + +## Papers + +1. **The Humble Programmer** + + Edsger W. Dijkstra, 1972 ([paper](http://dl.acm.org/citation.cfm?id=361591)) + +## Programming + +1. **Design Patterns: Elements of Reusable Object-Oriented Software** + + Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides, 1994 ([amazon](http://www.amazon.com/Design-Patterns-Elements-Reusable-Object-Oriented/dp/0201633612)) + +1. **Clean Code: A Handbook of Agile Software Craftsmanship** + + Robert C. "Uncle Bob" Martin, 2008 ([amazon](http://www.amazon.com/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882)) + +1. **Code Complete: A Practical Handbook of Software Construction**, 2nd Edition + + Steve McConnell, 2004 ([amazon](http://www.amazon.com/Code-Complete-Practical-Handbook-Construction/dp/0735619670)) + +1. **The Pragmatic Programmer: From Journeyman to Master** + + Andrew Hunt, David Thomas, 1999 ([amazon](http://www.amazon.com/Pragmatic-Programmer-Journeyman-Master/dp/020161622X)) + +1. **Working Effectively with Legacy Code** + + Michael Feathers, 2004 ([amazon](http://www.amazon.com/Working-Effectively-Legacy-Michael-Feathers/dp/0131177052)) + +1. **Eloquent Ruby** + + Russ Olsen, 2011 ([amazon](http://www.amazon.com/Eloquent-Ruby-Addison-Wesley-Professional/dp/0321584104)) + +1. **Domain-Driven Design: Tackling Complexity in the Heart of Software** + + Eric Evans, 2003 ([amazon](http://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software/dp/0321125215)) + +1. **How to Solve It: A New Aspect of Mathematical Method** + + Polya G. 1957 ([amazon](http://www.amazon.com/How-Solve-Mathematical-Princeton-Science/dp/069116407X)) + +1. **Software Creativity 2.0** + + Robert L. Glass, 2006 ([amazon](http://www.amazon.com/Software-Creativity-2-0-Robert-Glass/dp/0977213315)) + +1. **Object-Oriented Software Construction** + + Bertrand Meyer, 1997 ([amazon](http://www.amazon.com/Object-Oriented-Software-Construction-Book-CD-ROM/dp/0136291554)) + +1. **Refactoring: Improving the Design of Existing Code** + + Martin Fowler, Kent Beck, 1999 ([amazon](http://www.amazon.com/Refactoring-Improving-Design-Existing-Code/dp/0201485672)) + +1. **Test Driven Development: By Example** + + Kent Beck, 2002 ([amazon](http://www.amazon.com/Test-Driven-Development-Kent-Beck/dp/0321146530)) + +1. **Algorithms in C++: Fundamentals, Data Structure, Sorting, Searching** + + Robert Sedgewick, 1990 ([amazon](http://www.amazon.com/Algorithms-Parts-1-4-Fundamentals-Structure/dp/0201350882)) + +1. **Effective C++** + + Scott Mayers, 1996 ([amazon](http://www.amazon.com/Effective-Specific-Improve-Programs-Designs/dp/0321334876)) + +1. **Extreme Programming Explained: Embrace Change** + + Kent Beck, 1999 ([amazon](http://www.amazon.com/Extreme-Programming-Explained-Embrace-Change/dp/0321278658)) + +1. **The Art of Computer Programming** + + Donald E. Knuth, 1997 ([amazon](http://www.amazon.com/Computer-Programming-Volumes-1-4A-Boxed/dp/0321751043)) + +1. **Writing Efficient Programs** + + Jon Louis Bentley, 1982 ([amazon](http://www.amazon.com/Writing-Efficient-Programs-Prentice-Hall-Software/dp/013970244X)) + +1. **The Mythical Man-Month: Essays on Software Engineering** + + Frederick Phillips Brooks, 1975 ([amazon](http://www.amazon.com/Mythical-Man-Month-Essays-Software-Engineering/dp/0201006502)) + +1. **Peopleware: Productive Projects and Teams** 3rd Edition + + Tom DeMarco, Tim Lister, 2013 ([amazon](http://www.amazon.com/Peopleware-Productive-Projects-Teams-3rd/dp/0321934113)) + +1. **Principles Of Software Engineering Management** + + Tom Gilb, 1988 ([amazon](http://www.amazon.com/Principles-Software-Engineering-Management-Gilb/dp/0201192462)) + +## Other + +1. **Thinking, Fast and Slow** + + Daniel Kahneman, 2013 ([amazon](http://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555)) + +1. **The Social Animal** 11th Edition + + Elliot Aronson, 2011 ([amazon](http://www.amazon.com/Social-Animal-Elliot-Aronson/dp/1429233419)) + +1. **Influence: Science and Practice** 5th Edition + + Robert B. Cialdini, 2008 ([amazon](http://www.amazon.com/Influence-Practice-Robert-B-Cialdini/dp/0205609996)) + +1. **Getting to Yes: Negotiating Agreement Without Giving In** + + Roger Fisher, William L. Ury, Bruce Patton, 2011 ([amazon](http://www.amazon.com/Getting-Yes-Negotiating-Agreement-Without/dp/0143118757)) + +1. **How to Win Friends & Influence People** + + Dale Carnegie, 1981 ([amazon](http://www.amazon.com/How-Win-Friends-Influence-People/dp/0671027034)) diff --git a/doc/university/bookclub/index.md b/doc/university/bookclub/index.md new file mode 100644 index 00000000000..022a61f4429 --- /dev/null +++ b/doc/university/bookclub/index.md @@ -0,0 +1,19 @@ +# The GitLab Book Club + +The Book Club is a casual meet-up to read and discuss books we like. +We'll find a time that suits most, if not all. + +See the [book list](booklist.md) for additional recommendations. + +## Currently reading : Books about remote work + +1. **Remote: Office not required** + + David Heinemeier Hansson and Jason Fried, 2013 + ([amazon](http://www.amazon.co.uk/Remote-Required-David-Heinemeier-Hansson/dp/0091954673)) + +1. **The Year Without Pants** + + Scott Berkun, 2013 ([ScottBerkun.com](http://scottberkun.com/yearwithoutpants/)) + +Any other books you'd like to suggest? Edit this page and add them to the queue. diff --git a/doc/university/glossary/README.md b/doc/university/glossary/README.md index a86ff165f2e..cf836667fac 100644 --- a/doc/university/glossary/README.md +++ b/doc/university/glossary/README.md @@ -6,83 +6,87 @@ Please add any terms that you discover that you think would be useful for others ### 2FA -User authentication by combination of 2 different steps during login. This allows for more security. +User authentication by combination of 2 different steps during login. This allows for [more security](https://about.gitlab.com/handbook/security/). ### Access Levels -Process of selective restriction to create, view, modify or delete a resource based on a set of assigned permissions. -See, [GitLab's Permission Guidelines](http://doc.gitlab.com/ce/permissions/permissions.html) +Process of selective restriction to create, view, modify or delete a resource based on a set of assigned permissions. See [GitLab's Permission Guidelines](http://doc.gitlab.com/ce/permissions/permissions.html) ### Active Directory (AD) -A Microsoft based directory service for windows domain networks. It uses LDAP technology under the hood +A Microsoft-based [directory service](https://msdn.microsoft.com/en-us/library/bb742424.aspx) for windows domain networks. It uses LDAP technology under the hood. ### Agile -Building and delivering software in phases/parts rather than trying to build everything at once then delivering to the user/client. The later is known as a WaterFall model +Building and [delivering software](http://agilemethodology.org/) in phases/parts rather than trying to build everything at once then delivering to the user/client. The latter is known as the WaterFall model. ### Application Lifecycle Management (ALM) -Entire product lifecycle management process for an application. From requirements management, development and testing until deployment. +The entire product lifecycle management process for an application, from requirements management, development, and testing until deployment. GitLab has [advantages](https://docs.google.com/presentation/d/1vCU-NbZWz8NTNK8Vu3y4zGMAHb5DpC8PE5mHtw1PWfI/edit#slide=id.g72f2e4906_2_288) over both legacy and modern ALM tools. ### Artifactory -Version control for binaries. +A version control [system](https://www.jfrog.com/open-source/#os-arti) for non-text files. ### Artifacts -objects (usually binary and large) created by a build process +Objects (usually binary and large) created by a build process. These can include use cases, class diagrams, requirements and design documents. ### Atlassian -A company that develops software products for developers and project managers including Bitbucket, Jira, Hipchat, Confluence, Bamboo. See [Atlassian] (https://www.atlassian.com) +A [company](https://www.atlassian.com) that develops software products for developers and project managers including Bitbucket, Jira, Hipchat, Confluence, Bamboo. ### Audit Log -*** Needs definition here +Also called an [audit trail](https://en.wikipedia.org/wiki/Audit_trail), an audit log is a document that records an event in an IT system. ### Auto Defined User Group -User groups are a way of centralizing control over important management tasks, particularly access control and password policies. -A simple example of such groups are the users and the admins groups. -In most of the cases these groups are auto defined in terms of access, rules of usage, conditions to be part of, etc... +User groups are a way of centralizing control over important management tasks, particularly access control and password policies. A simple example of such groups are the users and the admins groups. +In most of the cases these groups are auto defined in terms of access, rules of usage, conditions to be part of, etc. ### Bamboo -Atlassian's CI tool similar to GitLab CI and Jenkins +Atlassian's CI tool similar to GitLab CI and Jenkins. ### Basic Subscription -Entry level subscription for GitLab EE currently available in packs of 10 see [Basic subscription](https://about.gitlab.com/pricing/) +Entry level [subscription](https://about.gitlab.com/pricing/) for GitLab EE currently available in packs of 10. ### Bitbucket -Atlassian's web hosting service for Git and Mercurial Projects i.e. GitLab.com competitor +Atlassian's web hosting service for Git and Mercurial Projects. Read about [migrating](https://docs.gitlab.com/ce/workflow/importing/import_projects_from_bitbucket.html) from BitBucket to a GitLab instance. ### Branch -A branch is a parallel version of a repository. Allows you to work on the repository without you affecting the "master" branch. Allows you to make changes without affecting the current "live" version. When you have made all your changes to your branch you can then merge to the master and to make the changes fo "live". +A branch is a parallel version of a repository. This allows you to work on the repository without affecting the "master" branch, and without affecting the current "live" version. When you have made all your changes to your branch you can then merge to the master. When your merge request is accepted your changes will be "live." ### Branded Login -Having your own logo on your GitLab instance login page instead of the GitLab logo. +Having your own logo on [your GitLab instance login page](https://docs.gitlab.com/ee/customization/branded_login_page.html) instead of the GitLab logo. + +### Build triggers +These protect your code base against breaks, for instance when a team is working on the same project. Learn about [setting up](https://docs.gitlab.com/ce/ci/triggers/README.html) build triggers. ### CEPH -is a distributed object store and file system designed to provide excellent performance, reliability and scalability. + A distributed object store and file [system](http://ceph.com/) designed to provide excellent performance, reliability and scalability. + +### ChatOps + +The ability to [initiate an action](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1412) from chat. ChatBots run in your chat application and give you the ability to do "anything" from chat. ### Clone -A copy of a repository stored on your machine that allows you to use your own editor without being online, but still tracks the changes made remotely. +A [copy](https://git-scm.com/docs/git-clone) of a repository stored on your machine that allows you to use your own editor without being online, but still tracks the changes made remotely. ### Code Review -Examination of a progam's code. The main aim is to maintain high standards quality of code that is being shipped. +Examination of a progam's code. The main aim is to maintain high quality standards of code that is being shipped. Merge requests [serve as a code review tool](https://about.gitlab.com/2014/09/29/gitlab-flow/) in GitLab. ### Code Snippet -A small amount of code. Usually for the purpose of showing other developers how -to do something specific or reproduce a problem. +A small amount of code, usually selected for the purpose of showing other developers how to do something specific or reproduce a problem. ### Collaborator @@ -90,31 +94,39 @@ Person with read and write access to a repository who has been invited by reposi ### Commit -Is a change (revision) to a file, and also creates an ID that allows you to see revision history and who made the changes. +A [change](https://git-scm.com/docs/git-commit) (revision) to a file that also creates an ID, allowing you to see revision history and the author of the changes. ### Community -Everyone who is using GitLab +[Everyone](https://about.gitlab.com/community/) who uses GitLab. ### Confluence -Atlassian's product for collaboration of documents and projects. +Atlassian's product for collaboration on documents and projects. -### Continuous Deivery +### Continuous Delivery -Continuous delivery is a series of practices designed to ensure that code can be rapidly and safely deployed to production by delivering every change to a production-like environment and ensuring business applications and services function as expected through rigorous automated testing. +A [software engineering approach](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/) in which continuous integration, automated testing, and automated deployment capabilities allow software to be developed and deployed rapidly, reliably and repeatedly with minimal human intervention. Still, the deployment to production is defined strategically and triggered manually. ### Continuous Deployment -Continuous deployment is the next step of continuous delivery: Every change that passes the automated tests is deployed to production automatically. +A [software development practice](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/) in which every code change goes through the entire pipeline and is put into production automatically, resulting in many production deployments every day. It does everything that Continuous Delivery does, but the process is fully automated, there's no human intervention at all. ### Continuous Integration -A process that involves adding new code commits to source code with the combined code being run on an automated test to ensure that the changes do not break the software. +A [software development practice](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/) in which you build and test software every time a developer pushes code to the application, and it happens several times a day. ### Contributor -Term used to a person contributing to an Open Source Project. +Term used for a person contributing to an open source project. + +### Conversational Development (ConvDev) + +A [natural evolution](https://about.gitlab.com/2016/09/14/gitlab-live-event-recap/) of software development that carries a conversation across functional groups throughout the development process, enabling developers to track the full path of development in a cohesive and intuitive way. ConvDev accelerates the development lifecycle by fostering collaboration and knowledge sharing from idea to production. + +### Cycle Time + +The time it takes to move from [idea to production](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#from-idea-to-production-with-gitlab). ### Data Centre @@ -122,41 +134,59 @@ Atlassian product for High Availability. ### Deploy Keys -An SSH key stored on the your server that grants access to a single GitLab repository. This is used by a GitLab runner to clone a project's code so that tests can be run against the checked out code. +A [SSH key](https://docs.gitlab.com/ce/gitlab-basics/create-your-ssh-keys.html)stored on your server that grants access to a single GitLab repository. This is used by a GitLab runner to clone a project's code so that tests can be run against the checked out code. ### Developer -For us (GitLab) this means a software developer, i.e. someone who makes software. It is also one of the levels of access in our multi level approval system. +For us at GitLab, this means a software developer, or someone who makes software. It is also one of the levels of access in our multi-level approval system. + +### DevOps + +The intersection of software engineering, quality assurance, and technology operations. Explore more DevOps topics in the [glossary by XebiaLabs](https://xebialabs.com/glossary/) ### Diff -Is the difference between two commits, or saved changes. This will also be shown visually after the changes. +The difference between two commits, or saved changes. This will also be shown visually after the changes. -### Docker +#### Directory -Containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. -This guarantees that it will always run the same, regardless of the environment it is running in. +A folder used for storing multiple files. + +### Docker Container Registry + +A [feature](https://docs.gitlab.com/ce/user/project/container_registry.html) of GitLab projects. Containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in. + +### Dynamic Environment + +### ElasticSearch + +Elasticsearch is a flexible, scalable and powerful search service. When [enabled](https://gitlab.com/help/integration/elasticsearch.md), it helps keep GitLab's search fast when dealing with a huge amount of data. + +### Emacs ### Fork -Your own copy of a repository that allows you to make changes to the repository without affecting the original. +Your [own copy](https://docs.gitlab.com/ce/workflow/forking_workflow.html) of a repository that allows you to make changes to the repository without affecting the original. ### Gerrit -A code review tool built on top of Git. +A code review [tool](https://www.gerritcodereview.com/) built on top of Git. + +### Git Attributes + +A [git attributes file](https://git-scm.com/docs/gitattributes) is a simple text file that gives attributes to pathnames. ### Git Hooks -Are scripts you can use to trigger actions at certain points. +[Scripts](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) you can use to trigger actions at certain points. ### GitHost.io -Is a single-tenant solution that provides GitLab CE or EE as a managed service. GitLab Inc. is responsible for -installing, updating, hosting, and backing up customers own private and secure GitLab instance. +A single-tenant solution that provides GitLab CE or EE as a managed service. GitLab Inc. is responsible for installing, updating, hosting, and backing up customers' own private and secure GitLab instance. ### GitHub -A web-based Git repository hosting service with an enterprise offering. Its main features are: issue tracking, pull request with code review, abundancy of integrations and wiki. As of April 2016, the service has over 14 million users. It offers free public repos, private repos and enterprise services are paid. +A web-based Git repository hosting service with an enterprise offering. Its main features are: issue tracking, pull request with code review, abundancy of integrations and wiki. It offers free public repos, private repos and enterprise services are paid. Read about [importing a project](https://docs.gitlab.com/ce/workflow/importing/import_projects_from_github.html) from GitHub to GitLab. ### GitLab CE @@ -164,51 +194,78 @@ Our free on Premise solution with >100,000 users ### GitLab CI -Our own Continuos Integration feature that is shipped with each instance +Our own Continuos Integration [feature](https://about.gitlab.com/gitlab-ci/) that is shipped with each instance ### GitLab EE -Our premium on premise solution that currently has Basic, Standard and Plus subscription packages with additional features and support. +Our premium on premise [solution](https://about.gitlab.com/features/#enterprise) that currently has Basic, Standard and Plus subscription packages with additional features and support. ### GitLab.com Our free SaaS for public and private repositories. +### GitLab Geo + +Allows you to replicate your GitLab instance to other geographical locations as a read-only fully operational version. It [can be used](https://docs.gitlab.com/ee/gitlab-geo/README.html) for cloning and fetching projects, in addition to reading any data. This will make working with large repositories over large distances much faster. + +### GitLab Pages +These allow you to [create websites](https://gitlab.com/help/pages/README.md) for your GitLab projects, groups, or user account. + ### Gitolite -Is basically an access layer that sits on top of Git. Users are granted access to repos via a simple config file and you as an admin only needs the users public SSH key and a username from the user. +An [access layer](https://git-scm.com/book/en/v1/Git-on-the-Server-Gitolite) that sits on top of Git. Users are granted access to repos via a simple config file. As an admin, you only need the users' public SSH key and a username. ### Gitorious -A web based hosting service for projects using Git. It was acquired by GitLab and we discontinued the service. [Gitorious Acquisition Blog Post](https://about.gitlab.com/2015/03/03/gitlab-acquires-gitorious/) +A web-based hosting service for projects using Git. It was acquired by GitLab and we discontinued the service. Read the[Gitorious Acquisition Blog Post](https://about.gitlab.com/2015/03/03/gitlab-acquires-gitorious/). + +### Go + +An open source programming [language](https://golang.org/). -### HADR +### GUI/ Git GUI -Sometimes written HA/DR. High Availability for Disaster Recovery. Usually refers to a strategy having a failover server in place in case the main server fails. +A portable [graphical interface](https://git-scm.com/docs/git-gui) to Git that allows users to make changes to their repository by making new commits, amending existing ones, creating branches, performing local merges, and fetching/pushing to remote repositories. + +### High Availability for Disaster Recovery (HADR) + +Sometimes written HA/DR, this usually refers to a strategy for having a failover server in place in case the main server fails. ### Hip Chat -Atlassian's real time chat application for teams. Competitor to Slack, RocketChat and MatterMost. +Atlassian's real time chat application for teams, Hip Chat is a competitor to Slack, RocketChat and MatterMost. ### High Availability -Refers to a system or component that is continuously operational for a desirably long length of time. Availability can be measured relative to "100% operational" or "never failing." +Refers to a [system or component](https://about.gitlab.com/high-availability/) that is continuously operational for a desirably long length of time. Availability can be measured relative to "100% operational" or "never failing." + +### Inner-sourcing + +The [use of](https://about.gitlab.com/2014/09/05/innersourcing-using-the-open-source-workflow-to-improve-collaboration-within-an-organization/) open source development techniques within the corporation. + +### Internet Relay Chat (IRC) + +An [application layer protocol](http://www.irchelp.org/) that facilitates communication in the form of text. ### Issue Tracker -A tool used to manage, organize, and maintain a list of issues, making it easier for an organization to manage. +A [tool](https://docs.gitlab.com/ee/integration/external-issue-tracker.html) used to manage, organize, and maintain a list of issues, making it easier for an organization to manage. ### Jenkins -An Open Source CI tool written using the Java programming language. Does the same job as GitLab CI, Bamboo, Travis CI. It is extremely popular. see [Jenkins](https://jenkins-ci.org/) +An Open Source CI tool written using the Java programming language. [Jenkins](https://jenkins-ci.org/) does the same job as GitLab CI, Bamboo, and Travis CI. It is extremely popular. ### Jira -Atlassian's project management software. i.e. a complex issue tracker. See[Jira](https://www.atlassian.com/software/jira) +Atlassian's [project management software](https://www.atlassian.com/software/jira), i.e. a complex issue tracker. GitLab [can be configured](https://docs.gitlab.com/ee/project_services/jira.html) to interact with JIRA Core either using an on-premise instance or the SaaS solution that Atlassian offers. + +### JUnit + +A testing framework for the Java programming language, [JUnit](http://junit.org/junit4/) has been important in the evolution of test-driven development. ### Kerberos -A network authentication protocol that uses secret-key cryptography for security. +A network authentication [protocol](http://web.mit.edu/kerberos/) that uses secret-key cryptography for security. ### Kubernetes @@ -216,23 +273,27 @@ An open source container cluster manager originally designed by Google. It's bas ### Labels -An identifier to describe a group of one or more specific file revisions +An [identifier](https://docs.gitlab.com/ce/user/project/labels.html) to describe a group of one or more specific file revisions. -### LDAP +### Lightweight Directory Access Protocol (LDAP) -Lightweight Directory Access Protocol - basically its a directory (electronic address book) with user information e.g. name, phone_number etc + A directory (electronic address book) with user information (e.g. name, phone_number etc.) ### LDAP User Authentication -Allowing GitLab to sign in people from an LDAP server i.e. Allow people whose names are on the electronic user directory server) to be able to use their LDAP accounts to login. +GitLab [integrates](https://docs.gitlab.com/ce/administration/auth/ldap.html) with LDAP to support user authentication. This enables GitLab to sign in people from an LDAP server (i.e., allowing people whose names are on the electronic user directory server to be able to use their LDAP accounts to login.) ### LDAP Group Sync Allows you to synchronize the members of a GitLab group with one or more LDAP groups. -### Git LFS +### Load Balancer -Git Large File Storage. A way to enable git to handle large binary files by using reference pointers within small text files to point to the large files. +A [device](https://en.wikipedia.org/wiki/Load_balancing_(computing)) that distributes network or application traffic across multiple servers. + +### Git Large File Storage (LFS) + +A way [to enable](https://about.gitlab.com/2015/11/23/announcing-git-lfs-support-in-gitlab/) git to handle large binary files by using reference pointers within small text files to point to the large files. Large files such as high resolution images and videos, audio files, and assets can be called from a remote server. ### Linux @@ -240,8 +301,7 @@ An operating system like Windows or OS X. It is mostly used by software develope ### Markdown -Is a lightweight markup language with plain text formatting syntax designed so that it can be converted to HTML and many other formats using a tool by the same name. -Markdown is often used to format readme files, for writing messages in online discussion forums, and to create rich text using a plain text editor. +A lightweight markup language with plain text formatting syntax designed so that it can be converted to HTML and many other formats using a tool by the same name. Markdown is often used to format readme files, for writing messages in online discussion forums, and to create rich text using a plain text editor. Checkout GitLab's [Markdown guide](https://gitlab.com/help/user/markdown.md). ### Maria DB @@ -249,193 +309,215 @@ A community developed fork/variation of MySQL. MySQL is owned by Oracle. ### Master -Name of the default branch in every git repository. +Name of the [default branch](https://git-scm.com/book/en/v1/Git-Branching-What-a-Branch-Is) in every git repository. + +### Mattermost + +An open source, self-hosted messaging alternative to Slack. View GitLab's Mattermost [feature](https://gitlab.com/gitlab-org/gitlab-mattermost). ### Mercurial -A free distributed version control system like Git. Think of it as a competitor to Git. +A free distributed version control system similar to and a competitor with Git. ### Merge -Takes changes from one branch, and applies them into another branch. +Takes changes from one branch, and [applies them](https://git-scm.com/docs/git-merge) into another branch. + +### Merge Conflict + +[Arises](https://about.gitlab.com/2016/09/06/resolving-merge-conflicts-from-the-gitlab-ui/) when a merge can't be performed cleanly between two versions of the same file. ### Meteor -A hip platform for building javascript apps.[Meteor] (https://www.meteor.com) +A [platform](https://www.meteor.com) for building javascript apps. ### Milestones -Allows you to track the progress on issues, and merge requests, which allows you to get a snapshot of the progress made. +Allow you to [organize issues](https://docs.gitlab.com/ce/workflow/milestones.html) and merge requests in GitLab into a cohesive group, optionally setting a due date. A common use is keeping track of an upcoming software version. Milestones are created per-project. ### Mirror Repositories -You can set up a project to automatically have its branches, tags, and commits updated from an upstream repository. This is useful when a repository you're interested in is located on a different server, and you want to be able to browse its content and its activity using the familiar GitLab interface. +A project that is setup to automatically have its branches, tags, and commits [updated from an upstream repository](https://docs.gitlab.com/ee/workflow/repository_mirroring.html). This is useful when a repository you're interested in is located on a different server, and you want to be able to browse its content and activity using the familiar GitLab interface. ### MIT License -A type of software license. It lets people do anything with your code with proper attribution and without warranty. It is the most common license for open source applications written in Ruby on Rails. GitLab CE is issued under this license. -This means, you can download the code, modify it as you want even build a new commercial product using the underlying code and its not illegal. The only condition is that there is no form of waranty provided by GitLab so whatever happens if you use the code is your own problem. - -### Mondo +A type of software license. It lets people do anything with your code with proper attribution and without warranty. It is the most common license for open source applications written in Ruby on Rails. GitLab CE is issued under this [license](https://docs.gitlab.com/ce/development/licensing.html). This means you can download the code, modify it as you want, and even build a new commercial product using the underlying code and it's not illegal. The only condition is that there is no form of warranty provided by GitLab so whatever happens when you use the code is your own problem. -*** Needs definition here +### Mondo Rescue -### Multi LDAP Server +A free disaster recovery [software](https://help.ubuntu.com/community/MondoMindi). -*** Needs definition here +### MySQL -### My SQL - -A relational database. Currently only supported if you are using EE. It is owned by Oracle. +A relational [database](http://www.mysql.com/) owned by Oracle. Currently only supported if you are using EE. ### Namespace -In computing, a namespace is a set of symbols that are used to organize objects of various kinds, so that these objects may be referred to by name. - -Prominent examples include: -- file systems are namespaces that assign names to files; -- programming languages organize their variables and subroutines in namespaces; -- computer networks and distributed systems assign names to resources, such as computers, printers, websites, (remote) files, etc. +A set of symbols that are used to organize objects of various kinds so that these objects may be referred to by name. Examples of namespaces in action include file systems that assign names to files; programming languages that organize their variables and subroutines in namespaces; and computer networks and distributed systems that assign names to resources, such as computers, printers, websites, (remote) files, etc. ### Nginx -(pronounced "engine x") is a web server. It can act as a reverse proxy server for HTTP, HTTPS, SMTP, POP3, and IMAP protocols, as well as a load balancer and an HTTP cache. +A web [server](https://www.nginx.com/resources/wiki/) (pronounced "engine x"). It can act as a reverse proxy server for HTTP, HTTPS, SMTP, POP3, and IMAP protocols, as well as a load balancer and an HTTP cache. -### oAuth +### OAuth -Is an open standard for authorization, commonly used as a way for Internet users to log into third party websites using their Microsoft, Google, Facebook or Twitter accounts without exposing their password. +An open standard for authorization, commonly used as a way for internet users to log into third party websites using their Microsoft, Google, Facebook or Twitter accounts without exposing their password. GitLab [is](https://docs.gitlab.com/ce/integration/oauth_provider.html) an OAuth2 authentication service provider. ### Omnibus Packages -Omnibus is a way to package the different services and tools required to run GitLab, so that users can install it without as much work. +A way to [package different services and tools](https://docs.gitlab.com/omnibus/) required to run GitLab, so that most developers can install it without laborious configuration. ### On Premise -On your own server. In GitLab, this refers to the ability to download GitLab EE/GitLab CE and host it on your own server rather than using GitLab.com which is hosted by GitLab Inc's servers. +On your own server. In GitLab, this [refers](https://about.gitlab.com/2015/02/12/why-ship-on-premises-in-the-saas-era/) to the ability to download GitLab EE/GitLab CE and host it on your own server rather than using GitLab.com, which is hosted by GitLab Inc's servers. + +### Open Core + +GitLab's [business model](https://about.gitlab.com/2016/07/20/gitlab-is-open-core-github-is-closed-source/). Coined by Andrew Lampitt in 2008, the [open core model](https://en.wikipedia.org/wiki/Open_core) primarily involves offering a "core" or feature-limited version of a software product as free and open-source software, while offering "commercial" versions or add-ons as proprietary software. ### Open Source Software -Software for which the original source code is freely available and may be redistributed and modified. +Software for which the original source code is freely [available](https://opensource.org/docs/osd) and may be redistributed and modified. GitLab prioritizes open source [stewardship](https://about.gitlab.com/2016/01/11/being-a-good-open-source-steward/). ### Owner -This is the most powerful person on a GitLab project. He has the permissions of all the other users plus the additional permission of being able to destroy i.e. delete the project +The most powerful person on a GitLab project. They have the permissions of all the other users plus the additional permission of being able to destroy (i.e. delete) the project. -### PaaS +### Platform as a Service (PaaS) -Typically referred to in regards to application development, it is a model in which a cloud provider delivers hardware and software tools to its users as a service +Typically referred to in regards to application development, PaaS is a model in which a cloud provider delivers hardware and software tools to its users as a service. ### Perforce -The company that produces Helix. A commercial, proprietary, centralised VCS well known for it's ability to version files of any size and type. They OEM a re-branded version of GitLab called "GitSwarm" that is tightly integrated with their "GitFusion" product, which in turn represents a portion of a Helix repository (called a depot) as a git repo +The company that produces Helix. A commercial, proprietary, centralised VCS well known for its ability to version files of any size and type. They OEM a re-branded version of GitLab called "GitSwarm" that is tightly integrated with their "GitFusion" product, which in turn represents a portion of a Helix repository (called a depot) as a git repo. ### Phabricator -Is a suite of web-based software development collaboration tools, including the Differential code review tool, the Diffusion repository browser, the Herald change monitoring tool, the Maniphest bug tracker and the Phriction wiki. Phabricator integrates with Git, Mercurial, and Subversion. +A suite of web-based software development collaboration tools, including the Differential code review tool, the Diffusion repository browser, the Herald change monitoring tool, the Maniphest bug tracker and the Phriction wiki. Phabricator integrates with Git, Mercurial, and Subversion. ### Piwik Analytics -An open source analytics software to help you analyze web traffic. It is similar to google analytics only that google analytics is not open source and information is stored by google while in Piwik the information is stored in your own server hence fully private. +An open source analytics software to help you analyze web traffic. It is similar to Google Analytics, except that the latter is not open source and information is stored by Google. In Piwik, the information is stored on your own server and hence is fully private. ### Plus Subscription -GitLab Premium EE subscription that includes training and dedicated Account Management and Service Engineer and complete support package [Plus subscription](https://about.gitlab.com/pricing/) +GitLab Premium EE [subscription](https://about.gitlab.com/pricing/) that includes training and dedicated Account Management and Service Engineer and complete support package. ### PostgreSQL -A relational database. Touted as the most advanced open source database. +An [object-relational](https://en.wikipedia.org/wiki/PostgreSQL) database. Touted as the most advanced open source database, it is one of two database management systems [supported by](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/database.md) GitLab, the other being MySQL. ### Protected Branches -A feature that protects branches from unauthorized pushes, force pushing or deletion. +A [feature](https://docs.gitlab.com/ce/user/project/protected_branches.html) that protects branches from unauthorized pushes, force pushing or deletion. ### Pull -Git command to synchronize the local repository with the remote repository, by fetching all remote changes and merging them into the local repository. +Git command to [synchronize](https://git-scm.com/docs/git-pull) the local repository with the remote repository, by fetching all remote changes and merging them into the local repository. ### Puppet -A popular devops automation tool +A popular DevOps [automation tool](https://puppet.com/product/how-puppet-works). ### Push -Git command to send commits from the local repository to the remote repository. +Git [command](https://git-scm.com/docs/git-push) to send commits from the local repository to the remote repository. Read about [advanced push rules](https://gitlab.com/help/pages/README.md) in GitLab. ### RE Read Only -Permissions to see a file and it's contents, but not change it +Permissions to see a file and its contents, but not change it. ### Rebase -Moves a branch from one commit to another. This allows you to re-write your project's history. +In addition to the merge, the [rebase](https://git-scm.com/book/en/v2/Git-Branching-Rebasing) is a main way to integrate changes from one branch into another. -### Git Repository +### (Git) Repository -Storage location of all files which are tracked by git. +A directory where Git [has been initiatlized](https://git-scm.com/book/en/v2/Git-Basics-Getting-a-Git-Repository) to start version controlling your files. The history of your work is stored here. A remote repository is not on your machine, but usually online (like on GitLab.com, for instance). The main remote repository is usually called "Origin." ### Requirements management -*** Needs definition here - -### Revision - -*** Needs definition here +Gives your distributed teams a single shared repository to collaborate and share requirements, understand their relationship to tests, and evaluate linked defects. It includes multiple, preconfigured requirement types. ### Revision Control -Also known as version control or source control, is the management of changes to documents, computer programs, large web sites, and other collections of information. Changes are usually identified by a number or letter code, termed the "revision number", "revision level", or simply "revision". +Also known as version control or source control, this is the management of changes to documents, computer programs, large web sites, and other collections of information. Changes are usually identified by a number or letter code, termed the "revision number," "revision level," or simply "revision." ### RocketChat -An open source chat application for teams. Very similar to Slack only that is is open-source. +An open source chat application for teams, RocketChat is very similar to Slack but it is also open-source. + +### Route Table + +A route table contains rules (called routes) that determine where network traffic is directed. Each [subnet in a VPC](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Route_Tables.html) must be associated with a route table. ### Runners -Actual build machines/containers that run/execute tests you have specified to be run on GitLab CI +Actual build machines/containers that [run and execute tests](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner) you have specified to be run on GitLab CI. + +### Sidekiq + +The background job processor GitLab [uses](https://docs.gitlab.com/ce/administration/troubleshooting/sidekiq.html) to asynchronously run tasks. -### SaaS +### Software as a service (SaaS) -Software as a service. Software is hosted centrally and accessed on-demand i.e. when you want to. This refers to GitLab.com in our scenario +Software that is hosted centrally and accessed on-demand (i.e. whenever you want to). This applies to GitLab.com. -### SCM +### Software Configuration Management (SCM) -Software Configuration Management. Often used by people when they mean Version Control +This term is often used by people when they mean "Version Control." ## Scrum -An Agile framework designed to help complete complex (typically) software projects. It's made up of several parts: product requirments backlog, sprint plannnig, sprint (development), sprint review, retrospec (analyzing the sprint). The goal is to end up with potentially shippable products. +An Agile [framework](https://www.scrum.org/Resources/What-is-Scrum) designed to typically help complete complex software projects. It's made up of several parts: product requirements backlog, sprint planning, sprint (development), sprint review, and retrospec (analyzing the sprint). The goal is to end up with potentially shippable products. ### Scrum Board The board used to track the status and progress of each of the sprint backlog items. +### Shell + +Terminal on Mac OSX, GitBash on Windows, or Linux Terminal on Linux. You [use git]() and make changes to GitLab projects in your shell. You [use git](https://docs.gitlab.com/ce/gitlab-basics/start-using-git.html) and make changes to GitLab projects in your shell. + +### Single-tenant + +The tenant purchases their own copy of the software and the software can be customized to meet the specific and needs of that customer. [GitHost.io](https://about.gitlab.com/handbook/positioning-faq/) is our provider of single-tenant 'managed cloud' GitLab instances. + ### Slack -Real time messaging app for teams. Used internally by GitLab +Real time messaging app for teams that is used internally by GitLab team members. GitLab users can enable [Slack integration](https://docs.gitlab.com/ce/project_services/slack.html) to trigger push, issue, and merge request events among others. ### Slave Servers -Also known as secondary servers. They help to spread the load over multiple machines, they also provide backups when the master/primary server crashes. +Also known as secondary servers, these help to spread the load over multiple machines. They also provide backups when the master/primary server crashes. ### Source Code -Program code as typed by a computer programmer. i.e. it has not yet been compiled/translated by the computer to machine language. +Program code as typed by a computer programmer (i.e. it has not yet been compiled/translated by the computer to machine language). ### SSH Key -A unique identifier of a computer. It is used to identify computers without the need for a password. e.g. On GitLab I have added the ssh key of all my work machines so that the GitLab instance knows that it can accept code pushes and pulls from this trusted machines whose keys are I have added. +A unique identifier of a computer. It is used to identify computers without the need for a password (e.g., On GitLab I have [added the ssh key](https://docs.gitlab.com/ce/gitlab-basics/create-your-ssh-keys.html) of all my work machines so that the GitLab instance knows that it can accept code pushes and pulls from this trusted machines whose keys are I have added.) -### SSO +### Single Sign On (SSO) -Single Sign On. An authentication process that allows you enter one username and password to access multiple applications. +An authentication process that allows you enter one username and password to access multiple applications. + +### Staging Area + +[Staging occurs](https://git-scm.com/book/en/v2/Getting-Started-Git-Basics) before the commit process in git. The staging area is a file, generally contained in your Git directory, that stores information about what will go into your next commit. It’s sometimes referred to as the “index."" ### Standard Subscription -Our mid range EE subscription that includes 24/7 support, support for High Availability [Standard Subscription](https://about.gitlab.com/pricing/) +Our mid range EE subscription that includes 24/7 support and support for High Availability [Standard Subscription](https://about.gitlab.com/pricing/). ### Stash -Atlassian's Git On-Premises solution. Think of it as Atlassian's GitLab EE. It is now known as BitBucket Server. +Atlassian's Git on-premise solution. Think of it as Atlassian's GitLab EE, now known as BitBucket Server. + +### Static Site Generators (SSGs) + +A [software](https://wiki.python.org/moin/StaticSiteGenerator) that takes some text and templates as input and produces html files on the output. ### Subversion @@ -443,40 +525,65 @@ Non-proprietary, centralized version control system. ### Sudo -A program that allows you to perform superuser/administrator actions on Unix Operating Systems e.g. Linux, OS X. It actually stands for 'superuser do' +A program that allows you to perform superuser/administrator actions on Unix Operating Systems (e.g., Linux, OS X.) It actually stands for 'superuser do.' -### SVN +### Subversion (SVN) -Abbreviation for Subversion. +An open source version control system. Read about [migrating from SVN](https://docs.gitlab.com/ce/workflow/importing/migrating_from_svn.html) to GitLab using SubGit. ### Tag -Represents a version of a particular branch at a moment in time. +[Represents](https://docs.gitlab.com/ce/api/tags.html) a version of a particular branch at a moment in time. ### Tool Stack -Set of tools used in a process to achieve a common outcome. E.g. set of tools used in Application Lifecycle Management. +The set of tools used in a process to achieve a common outcome (e.g. set of tools used in Application Lifecycle Management). ### Trac -An Open Source project management and bug tracking web application. +An open source project management and bug tracking web [application](https://trac.edgewall.org/). + +### Untracked files + +New files that Git has not [been told](https://git-scm.com/book/en/v2/Git-Basics-Recording-Changes-to-the-Repository) to track previously. ### User Anyone interacting with the software. -### VCS +### Version Control Software (VCS) + +Version control is a system that records changes to a file or set of files over time so that you can recall specific versions later. VCS [has evolved](https://docs.google.com/presentation/d/16sX7hUrCZyOFbpvnrAFrg6tVO5_yT98IgdAqOmXwBho/edit#slide=id.gd69537a19_0_32) from local version control systems, to centralized version control systems, to the present distributed version control systems like Git, Mercurial, Bazaar, and Darcs. + +### Virtual Private Cloud (VPC) + +An on demand configurable pool of shared computing resources allocated within a public cloud environment, providing some isolation between the different users using the resources. GitLab users need to create a new Amazon VPC in order to [setup High Availability](https://docs.gitlab.com/ce/university/high-availability/aws/). -Version Control Software +### Virtual private server (VPS) + +A [virtual machine](https://en.wikipedia.org/wiki/Virtual_private_server) sold as a service by an Internet hosting service. A VPS runs its own copy of an operating system, and customers have superuser-level access to that operating system instance, so they can install almost any software that runs on that OS. + +### VM Instance + +In object-oriented programming, an [instance](http://stackoverflow.com/questions/20461907/what-is-meaning-of-instance-in-programming) is a specific realization of any object. An object may be varied in a number of ways. Each realized variation of that object is an instance. Therefore, a VM instance is an instance of a virtual machine, which is an emulation of a computer system. ### Waterfall -A model of building software that involves collecting all requirements from the customer, then building and refining all the requirements and finally delivering the COMPLETE software to the customer that meets all the requirements specified by the customer +A [model](http://www.umsl.edu/~hugheyd/is6840/waterfall.html) of building software that involves collecting all requirements from the customer, then building and refining all the requirements and finally delivering the complete software to the customer that meets all the requirements they specified. ### Webhooks -A way for for an app to provide other applications with real-time information. e.g. send a message to a slack channel when a commit is pushed +A way for for an app to [provide](https://docs.gitlab.com/ce/web_hooks/web_hooks.html) other applications with real-time information (e.g., send a message to a slack channel when a commit is pushed.) Read about setting up [custom git hooks](https://gitlab.com/help/administration/custom_hooks.md) for when webhooks are insufficient. ### Wiki -A website/system that allows for collaborative editing of its content by the users. In programming, they usually contain documentation of how to use the software +A [website/system](http://www.wiki.com/) that allows for collaborative editing of its content by the users. In programming, wikis usually contain documentation of how to use the software. + +### Working Tree + +[Consists of files](http://stackoverflow.com/questions/3689838/difference-between-head-working-tree-index-in-git) that you are currently working on. + +### YAML + +A human-readable data serialization [language](http://www.yaml.org/about.html) that takes concepts from programming languages such as C, Perl, and Python, and ideas from XML and the data format of electronic mail. + diff --git a/doc/update/8.0-to-8.1.md b/doc/update/8.0-to-8.1.md index d57c0d0674d..bfb83cf79b1 100644 --- a/doc/update/8.0-to-8.1.md +++ b/doc/update/8.0-to-8.1.md @@ -99,6 +99,10 @@ sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS # Update init.d script sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab ``` + +For Ubuntu 16.04.1 LTS: + + sudo systemctl daemon-reload ### 7. Update configuration files diff --git a/doc/update/8.1-to-8.2.md b/doc/update/8.1-to-8.2.md index 46dfa2232b4..7f36ce00e96 100644 --- a/doc/update/8.1-to-8.2.md +++ b/doc/update/8.1-to-8.2.md @@ -116,6 +116,10 @@ sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS # Update init.d script sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab ``` + +For Ubuntu 16.04.1 LTS: + + sudo systemctl daemon-reload ### 7. Update configuration files diff --git a/doc/update/8.10-to-8.11.md b/doc/update/8.10-to-8.11.md index b058f8e2a03..119c5f475e4 100644 --- a/doc/update/8.10-to-8.11.md +++ b/doc/update/8.10-to-8.11.md @@ -22,7 +22,9 @@ sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production ### 3. Update Ruby -If you are you running Ruby 2.1.x, you do not _need_ to upgrade Ruby yet, but you should note that support for 2.1.x is deprecated and we will require 2.3.x in 8.13. It's strongly recommended that you upgrade as soon as possible. +We will continue supporting Ruby < 2.3 for the time being but we recommend you +upgrade to Ruby 2.3 if you're running a source installation, as this is the same +version that ships with our Omnibus package. You can check which version you are running with `ruby -v`. @@ -156,6 +158,10 @@ See [smtp_settings.rb.sample] as an example. Ensure you're still up-to-date with the latest init script changes: sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab + +For Ubuntu 16.04.1 LTS: + + sudo systemctl daemon-reload ### 9. Start application diff --git a/doc/update/8.11-to-8.12.md b/doc/update/8.11-to-8.12.md index 076696f565b..cddfa7e3e01 100644 --- a/doc/update/8.11-to-8.12.md +++ b/doc/update/8.11-to-8.12.md @@ -22,7 +22,9 @@ sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production ### 3. Update Ruby -If you are you running Ruby 2.1.x, you do not _need_ to upgrade Ruby yet, but you should note that support for 2.1.x is deprecated and we will require 2.3.x in 8.13. It's strongly recommended that you upgrade as soon as possible. +We will continue supporting Ruby < 2.3 for the time being but we recommend you +upgrade to Ruby 2.3 if you're running a source installation, as this is the same +version that ships with our Omnibus package. You can check which version you are running with `ruby -v`. @@ -70,7 +72,7 @@ sudo -u git -H git checkout 8-12-stable-ee ```bash cd /home/git/gitlab-shell sudo -u git -H git fetch --all --tags -sudo -u git -H git checkout v3.6.0 +sudo -u git -H git checkout v3.6.1 ``` ### 6. Update gitlab-workhorse @@ -164,6 +166,10 @@ See [smtp_settings.rb.sample] as an example. Ensure you're still up-to-date with the latest init script changes: sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab + +For Ubuntu 16.04.1 LTS: + + sudo systemctl daemon-reload ### 9. Start application diff --git a/doc/update/8.12-to-8.13.md b/doc/update/8.12-to-8.13.md new file mode 100644 index 00000000000..8940d14559b --- /dev/null +++ b/doc/update/8.12-to-8.13.md @@ -0,0 +1,205 @@ +# From 8.12 to 8.13 + +Make sure you view this update guide from the tag (version) of GitLab you would +like to install. In most cases this should be the highest numbered production +tag (without rc in it). You can select the tag in the version dropdown at the +top left corner of GitLab (below the menu bar). + +If the highest number stable branch is unclear please check the +[GitLab Blog](https://about.gitlab.com/blog/archives.html) for installation +guide links by version. + +### 1. Stop server + + sudo service gitlab stop + +### 2. Backup + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +### 3. Update Ruby + +We will continue supporting Ruby < 2.3 for the time being but we recommend you +upgrade to Ruby 2.3 if you're running a source installation, as this is the same +version that ships with our Omnibus package. + +You can check which version you are running with `ruby -v`. + +Download and compile Ruby: + +```bash +mkdir /tmp/ruby && cd /tmp/ruby +curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.1.tar.gz +echo 'c39b4001f7acb4e334cb60a0f4df72d434bef711 ruby-2.3.1.tar.gz' | shasum --check - && tar xzf ruby-2.3.1.tar.gz +cd ruby-2.3.1 +./configure --disable-install-rdoc +make +sudo make install +``` + +Install Bundler: + +```bash +sudo gem install bundler --no-ri --no-rdoc +``` + +### 4. Get latest code + +```bash +sudo -u git -H git fetch --all +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically +``` + +For GitLab Community Edition: + +```bash +sudo -u git -H git checkout 8-13-stable +``` + +OR + +For GitLab Enterprise Edition: + +```bash +sudo -u git -H git checkout 8-13-stable-ee +``` + +### 5. Update gitlab-shell + +```bash +cd /home/git/gitlab-shell +sudo -u git -H git fetch --all --tags +sudo -u git -H git checkout v3.6.3 +``` + +### 6. Update gitlab-workhorse + +Install and compile gitlab-workhorse. This requires +[Go 1.5](https://golang.org/dl) which should already be on your system from +GitLab 8.1. + +```bash +cd /home/git/gitlab-workhorse +sudo -u git -H git fetch --all +sudo -u git -H git checkout v0.8.5 +sudo -u git -H make +``` + +### 7. Install libs, migrations, etc. + +```bash +cd /home/git/gitlab + +# MySQL installations (note: the line below states '--without postgres') +sudo -u git -H bundle install --without postgres development test --deployment + +# PostgreSQL installations (note: the line below states '--without mysql') +sudo -u git -H bundle install --without mysql development test --deployment + +# Optional: clean up old gems +sudo -u git -H bundle clean + +# Run database migrations +sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + +# Clean up assets and cache +sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production +``` + +### 8. Update configuration files + +#### New configuration options for `gitlab.yml` + +There are new configuration options available for [`gitlab.yml`](config/gitlab.yml.example). View them with the command below and apply them manually to your current `gitlab.yml`: + +```sh +git diff origin/8-12-stable:config/gitlab.yml.example origin/8-13-stable:config/gitlab.yml.example +``` + +#### Git configuration + +Configure Git to generate packfile bitmaps (introduced in Git 2.0) on +the GitLab server during `git gc`. + +```sh +sudo -u git -H git config --global repack.writeBitmaps true +``` + +#### Nginx configuration + +Ensure you're still up-to-date with the latest NGINX configuration changes: + +```sh +# For HTTPS configurations +git diff origin/8-12-stable:lib/support/nginx/gitlab-ssl origin/8-13-stable:lib/support/nginx/gitlab-ssl + +# For HTTP configurations +git diff origin/8-12-stable:lib/support/nginx/gitlab origin/8-13-stable:lib/support/nginx/gitlab +``` + +If you are using Apache instead of NGINX please see the updated [Apache templates]. +Also note that because Apache does not support upstreams behind Unix sockets you +will need to let gitlab-workhorse listen on a TCP port. You can do this +via [/etc/default/gitlab]. + +[Apache templates]: https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache +[/etc/default/gitlab]: https://gitlab.com/gitlab-org/gitlab-ce/blob/8-13-stable/lib/support/init.d/gitlab.default.example#L38 + +#### SMTP configuration + +If you're installing from source and use SMTP to deliver mail, you will need to add the following line +to config/initializers/smtp_settings.rb: + +```ruby +ActionMailer::Base.delivery_method = :smtp +``` + +See [smtp_settings.rb.sample] as an example. + +[smtp_settings.rb.sample]: https://gitlab.com/gitlab-org/gitlab-ce/blob/8-13-stable/config/initializers/smtp_settings.rb.sample#L13 + +#### Init script + +Ensure you're still up-to-date with the latest init script changes: + + sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab + +For Ubuntu 16.04.1 LTS: + + sudo systemctl daemon-reload + +### 9. Start application + + sudo service gitlab start + sudo service nginx restart + +### 10. Check application status + +Check if GitLab and its environment are configured correctly: + + sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production + +To make sure you didn't miss anything run a more thorough check: + + sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production + +If all items are green, then congratulations, the upgrade is complete! + +## Things went south? Revert to previous version (8.12) + +### 1. Revert the code to the previous version + +Follow the [upgrade guide from 8.11 to 8.12](8.11-to-8.12.md), except for the +database migration (the backup is already migrated to the previous version). + +### 2. Restore from the backup + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` + +If you have more than one backup `*.tar` file(s) please add `BACKUP=timestamp_of_backup` to the command above. diff --git a/doc/update/8.2-to-8.3.md b/doc/update/8.2-to-8.3.md index 9f5c6c4dc84..dd3fdafd8d1 100644 --- a/doc/update/8.2-to-8.3.md +++ b/doc/update/8.2-to-8.3.md @@ -158,6 +158,10 @@ it where the 'public' directory of GitLab is. cd /home/git/gitlab sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab ``` + +For Ubuntu 16.04.1 LTS: + + sudo systemctl daemon-reload ### 8. Use Redis v2.8.0+ diff --git a/doc/update/8.3-to-8.4.md b/doc/update/8.3-to-8.4.md index 9f6517d9487..e62d894609a 100644 --- a/doc/update/8.3-to-8.4.md +++ b/doc/update/8.3-to-8.4.md @@ -98,6 +98,10 @@ We updated the init script for GitLab in order to set a specific PATH for gitlab cd /home/git/gitlab sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab ``` + +For Ubuntu 16.04.1 LTS: + + sudo systemctl daemon-reload ### 8. Start application diff --git a/doc/update/8.4-to-8.5.md b/doc/update/8.4-to-8.5.md index 0cb137a03cc..678cc69d773 100644 --- a/doc/update/8.4-to-8.5.md +++ b/doc/update/8.4-to-8.5.md @@ -119,6 +119,10 @@ via [/etc/default/gitlab]. Ensure you're still up-to-date with the latest init script changes: sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab + +For Ubuntu 16.04.1 LTS: + + sudo systemctl daemon-reload ### 8. Start application diff --git a/doc/update/8.5-to-8.6.md b/doc/update/8.5-to-8.6.md index 6267f14eba4..a76346516b9 100644 --- a/doc/update/8.5-to-8.6.md +++ b/doc/update/8.5-to-8.6.md @@ -138,6 +138,10 @@ via [/etc/default/gitlab]. Ensure you're still up-to-date with the latest init script changes: sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab + +For Ubuntu 16.04.1 LTS: + + sudo systemctl daemon-reload ### 9. Start application diff --git a/doc/update/8.6-to-8.7.md b/doc/update/8.6-to-8.7.md index cb66ef920bb..05ef4e61759 100644 --- a/doc/update/8.6-to-8.7.md +++ b/doc/update/8.6-to-8.7.md @@ -127,6 +127,10 @@ via [/etc/default/gitlab]. Ensure you're still up-to-date with the latest init script changes: sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab + +For Ubuntu 16.04.1 LTS: + + sudo systemctl daemon-reload ### 8. Start application diff --git a/doc/update/8.7-to-8.8.md b/doc/update/8.7-to-8.8.md index 32906650f6f..8ce434e5f78 100644 --- a/doc/update/8.7-to-8.8.md +++ b/doc/update/8.7-to-8.8.md @@ -127,6 +127,10 @@ via [/etc/default/gitlab]. Ensure you're still up-to-date with the latest init script changes: sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab + +For Ubuntu 16.04.1 LTS: + + sudo systemctl daemon-reload ### 8. Start application diff --git a/doc/update/8.8-to-8.9.md b/doc/update/8.8-to-8.9.md index f078a2bece5..aa077316bbe 100644 --- a/doc/update/8.8-to-8.9.md +++ b/doc/update/8.8-to-8.9.md @@ -156,6 +156,10 @@ See [smtp_settings.rb.sample] as an example. Ensure you're still up-to-date with the latest init script changes: sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab + +For Ubuntu 16.04.1 LTS: + + sudo systemctl daemon-reload ### 9. Start application diff --git a/doc/update/8.9-to-8.10.md b/doc/update/8.9-to-8.10.md index a057a423e61..bb2c79fbb84 100644 --- a/doc/update/8.9-to-8.10.md +++ b/doc/update/8.9-to-8.10.md @@ -156,6 +156,10 @@ See [smtp_settings.rb.sample] as an example. Ensure you're still up-to-date with the latest init script changes: sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab + +For Ubuntu 16.04.1 LTS: + + sudo systemctl daemon-reload ### 9. Start application diff --git a/doc/user/admin_area/monitoring/health_check.md b/doc/user/admin_area/monitoring/health_check.md new file mode 100644 index 00000000000..eac57bc3de4 --- /dev/null +++ b/doc/user/admin_area/monitoring/health_check.md @@ -0,0 +1,66 @@ +# Health Check + +> [Introduced][ce-3888] in GitLab 8.8. + +GitLab provides a health check endpoint for uptime monitoring on the `health_check` web +endpoint. The health check reports on the overall system status based on the status of +the database connection, the state of the database migrations, and the ability to write +and access the cache. This endpoint can be provided to uptime monitoring services like +[Pingdom][pingdom], [Nagios][nagios-health], and [NewRelic][newrelic-health]. + +## Access Token + +An access token needs to be provided while accessing the health check endpoint. The current +accepted token can be found on the `admin/health_check` page of your GitLab instance. + + + +The access token can be passed as a URL parameter: + +``` +https://gitlab.example.com/health_check.json?token=ACCESS_TOKEN +``` + +or as an HTTP header: + +```bash +curl --header "TOKEN: ACCESS_TOKEN" https://gitlab.example.com/health_check.json +``` + +## Using the Endpoint + +Once you have the access token, health information can be retrieved as plain text, JSON, +or XML using the `health_check` endpoint: + +- `https://gitlab.example.com/health_check?token=ACCESS_TOKEN` +- `https://gitlab.example.com/health_check.json?token=ACCESS_TOKEN` +- `https://gitlab.example.com/health_check.xml?token=ACCESS_TOKEN` + +You can also ask for the status of specific services: + +- `https://gitlab.example.com/health_check/cache.json?token=ACCESS_TOKEN` +- `https://gitlab.example.com/health_check/database.json?token=ACCESS_TOKEN` +- `https://gitlab.example.com/health_check/migrations.json?token=ACCESS_TOKEN` + +For example, the JSON output of the following health check: + +```bash +curl --header "TOKEN: ACCESS_TOKEN" https://gitlab.example.com/health_check.json +``` + +would be like: + +``` +{"healthy":true,"message":"success"} +``` + +## Status + +On failure, the endpoint will return a `500` HTTP status code. On success, the endpoint +will return a valid successful HTTP status code, and a `success` message. Ideally your +uptime monitoring should look for the success message. + +[ce-3888]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3888 +[pingdom]: https://www.pingdom.com +[nagios-health]: https://nagios-plugins.org/doc/man/check_http.html +[newrelic-health]: https://docs.newrelic.com/docs/alerts/alert-policies/downtime-alerts/availability-monitoring diff --git a/doc/monitoring/img/health_check_token.png b/doc/user/admin_area/monitoring/img/health_check_token.png Binary files differindex 2d7c82a65a8..2d7c82a65a8 100644 --- a/doc/monitoring/img/health_check_token.png +++ b/doc/user/admin_area/monitoring/img/health_check_token.png diff --git a/doc/user/markdown.md b/doc/user/markdown.md index 56e5b802a52..7a7a0b864bd 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -501,6 +501,10 @@ There are two ways to create links, inline-style and reference-style. [I'm a reference-style link][Arbitrary case-insensitive reference text] [I'm a relative reference to a repository file](LICENSE) + + [I am an absolute reference within the repository](/doc/user/markdown.md) + + [I link to the Milestones page](/../milestones) [You can use numbers for reference-style link definitions][1] @@ -518,6 +522,10 @@ There are two ways to create links, inline-style and reference-style. [I'm a relative reference to a repository file](LICENSE)[^1] +[I am an absolute reference within the repository](/doc/user/markdown.md) + +[I link to the Milestones page](/../milestones) + [You can use numbers for reference-style link definitions][1] Or leave it empty and use the [link text itself][] diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 21fd450ec72..51a7fc9eaf7 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -26,6 +26,7 @@ The following table depicts the various user permission levels in a project. | See a list of builds | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | | See a build log | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | | Download and browse build artifacts | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | +| View wiki pages | ✓ | ✓ | ✓ | ✓ | ✓ | | Pull project code | | ✓ | ✓ | ✓ | ✓ | | Download project | | ✓ | ✓ | ✓ | ✓ | | Create code snippets | | ✓ | ✓ | ✓ | ✓ | @@ -34,6 +35,7 @@ The following table depicts the various user permission levels in a project. | See a commit status | | ✓ | ✓ | ✓ | ✓ | | See a container registry | | ✓ | ✓ | ✓ | ✓ | | See environments | | ✓ | ✓ | ✓ | ✓ | +| See a list of merge requests | | ✓ | ✓ | ✓ | ✓ | | Manage/Accept merge requests | | | ✓ | ✓ | ✓ | | Create new merge request | | | ✓ | ✓ | ✓ | | Create new branches | | | ✓ | ✓ | ✓ | diff --git a/doc/user/project/container_registry.md b/doc/user/project/container_registry.md new file mode 100644 index 00000000000..b205fea2c40 --- /dev/null +++ b/doc/user/project/container_registry.md @@ -0,0 +1,253 @@ +# GitLab Container Registry + +> [Introduced][ce-4040] in GitLab 8.8. + +--- + +> **Note** +Docker Registry manifest `v1` support was added in GitLab 8.9 to support Docker +versions earlier than 1.10. +> +This document is about the user guide. To learn how to enable GitLab Container +Registry across your GitLab instance, visit the +[administrator documentation](../../administration/container_registry.md). + +With the Docker Container Registry integrated into GitLab, every project can +have its own space to store its Docker images. + +You can read more about Docker Registry at https://docs.docker.com/registry/introduction/. + +--- + +## Enable the Container Registry for your project + +1. First, ask your system administrator to enable GitLab Container Registry + following the [administration documentation](../../administration/container_registry.md). + If you are using GitLab.com, this is enabled by default so you can start using + the Registry immediately. + +1. Go to your project's settings and enable the **Container Registry** feature + on your project. For new projects this might be enabled by default. For + existing projects (prior GitLab 8.8), you will have to explicitly enable it. + +  + +1. Hit **Save changes** for the changes to take effect. You should now be able + to see the **Registry** link in the project menu. + +  + +## Build and push images + +If you visit the **Registry** link under your project's menu, you can see the +explicit instructions to login to the Container Registry using your GitLab +credentials. + +For example if the Registry's URL is `registry.example.com`, the you should be +able to login with: + +``` +docker login registry.example.com +``` + +Building and publishing images should be a straightforward process. Just make +sure that you are using the Registry URL with the namespace and project name +that is hosted on GitLab: + +``` +docker build -t registry.example.com/group/project . +docker push registry.example.com/group/project +``` + +Your image will be named after the following scheme: + +``` +<registry URL>/<namespace>/<project> +``` + +As such, the name of the image is unique, but you can differentiate the images +using tags. + +## Use images from GitLab Container Registry + +To download and run a container from images hosted in GitLab Container Registry, +use `docker run`: + +``` +docker run [options] registry.example.com/group/project [arguments] +``` + +For more information on running Docker containers, visit the +[Docker documentation][docker-docs]. + +## Control Container Registry from within GitLab + +GitLab offers a simple Container Registry management panel. Go to your project +and click **Registry** in the project menu. + +This view will show you all tags in your project and will easily allow you to +delete them. + + + +## Build and push images using GitLab CI + +> **Note:** +This feature requires GitLab 8.8 and GitLab Runner 1.2. + +Make sure that your GitLab Runner is configured to allow building Docker images by +following the [Using Docker Build](../ci/docker/using_docker_build.md) +and [Using the GitLab Container Registry documentation](../ci/docker/using_docker_build.md#using-the-gitlab-container-registry). + +## Limitations + +In order to use a container image from your private project as an `image:` in +your `.gitlab-ci.yml`, you have to follow the +[Using a private Docker Registry][private-docker] +documentation. This workflow will be simplified in the future. + +## Troubleshooting the GitLab Container Registry + +### Basic Troubleshooting + +1. Check to make sure that the system clock on your Docker client and GitLab server have + been synchronized (e.g. via NTP). + +2. If you are using an S3-backed Registry, double check that the IAM + permissions and the S3 credentials (including region) are correct. See [the + sample IAM policy](https://docs.docker.com/registry/storage-drivers/s3/) + for more details. + +3. Check the Registry logs (e.g. `/var/log/gitlab/registry/current`) and the GitLab production logs + for errors (e.g. `/var/log/gitlab/gitlab-rails/production.log`). You may be able to find clues + there. + +### Advanced Troubleshooting + +>**NOTE:** The following section is only recommended for experts. + +Sometimes it's not obvious what is wrong, and you may need to dive deeper into +the communication between the Docker client and the Registry to find out +what's wrong. We will use a concrete example in the past to illustrate how to +diagnose a problem with the S3 setup. + +#### Unexpected 403 error during push + +A user attempted to enable an S3-backed Registry. The `docker login` step went +fine. However, when pushing an image, the output showed: + +``` +The push refers to a repository [s3-testing.myregistry.com:4567/root/docker-test] +dc5e59c14160: Pushing [==================================================>] 14.85 kB +03c20c1a019a: Pushing [==================================================>] 2.048 kB +a08f14ef632e: Pushing [==================================================>] 2.048 kB +228950524c88: Pushing 2.048 kB +6a8ecde4cc03: Pushing [==> ] 9.901 MB/205.7 MB +5f70bf18a086: Pushing 1.024 kB +737f40e80b7f: Waiting +82b57dbc5385: Waiting +19429b698a22: Waiting +9436069b92a3: Waiting +error parsing HTTP 403 response body: unexpected end of JSON input: "" +``` + +This error is ambiguous, as it's not clear whether the 403 is coming from the +GitLab Rails application, the Docker Registry, or something else. In this +case, since we know that since the login succeeded, we probably need to look +at the communication between the client and the Registry. + +The REST API between the Docker client and Registry is [described +here](https://docs.docker.com/registry/spec/api/). Normally, one would just +use Wireshark or tcpdump to capture the traffic and see where things went +wrong. However, since all communication between Docker clients and servers +are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even +if you know the private key. What can we do instead? + +One way would be to disable HTTPS by setting up an [insecure +Registry](https://docs.docker.com/registry/insecure/). This could introduce a +security hole and is only recommended for local testing. If you have a +production system and can't or don't want to do this, there is another way: +use mitmproxy, which stands for Man-in-the-Middle Proxy. + +#### mitmproxy + +[mitmproxy](https://mitmproxy.org/) allows you to place a proxy between your +client and server to inspect all traffic. One wrinkle is that your system +needs to trust the mitmproxy SSL certificates for this to work. + +The following installation instructions assume you are running Ubuntu: + +1. Install mitmproxy (see http://docs.mitmproxy.org/en/stable/install.html) +1. Run `mitmproxy --port 9000` to generate its certificates. + Enter <kbd>CTRL</kbd>-<kbd>C</kbd> to quit. +1. Install the certificate from `~/.mitmproxy` to your system: + + ```sh + sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt + sudo update-ca-certificates + ``` + +If successful, the output should indicate that a certificate was added: + +```sh +Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done. +Running hooks in /etc/ca-certificates/update.d....done. +``` + +To verify that the certificates are properly installed, run: + +```sh +mitmproxy --port 9000 +``` + +This will run mitmproxy on port `9000`. In another window, run: + +```sh +curl --proxy http://localhost:9000 https://httpbin.org/status/200 +``` + +If everything is setup correctly, you will see information on the mitmproxy window and +no errors from the curl commands. + +#### Running the Docker daemon with a proxy + +For Docker to connect through a proxy, you must start the Docker daemon with the +proper environment variables. The easiest way is to shutdown Docker (e.g. `sudo initctl stop docker`) +and then run Docker by hand. As root, run: + +```sh +export HTTP_PROXY="http://localhost:9000" +export HTTPS_PROXY="https://localhost:9000" +docker daemon --debug +``` + +This will launch the Docker daemon and proxy all connections through mitmproxy. + +#### Running the Docker client + +Now that we have mitmproxy and Docker running, we can attempt to login and push +a container image. You may need to run as root to do this. For example: + +```sh +docker login s3-testing.myregistry.com:4567 +docker push s3-testing.myregistry.com:4567/root/docker-test +``` + +In the example above, we see the following trace on the mitmproxy window: + + + +The above image shows: + +* The initial PUT requests went through fine with a 201 status code. +* The 201 redirected the client to the S3 bucket. +* The HEAD request to the AWS bucket reported a 403 Unauthorized. + +What does this mean? This strongly suggests that the S3 user does not have the right +[permissions to perform a HEAD request](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html). +The solution: check the [IAM permissions again](https://docs.docker.com/registry/storage-drivers/s3/). +Once the right permissions were set, the error will go away. + +[ce-4040]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4040 +[docker-docs]: https://docs.docker.com/engine/userguide/intro/ +[private-docker]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/configuration/advanced-configuration.md#using-a-private-docker-registry diff --git a/doc/user/project/cycle_analytics.md b/doc/user/project/cycle_analytics.md index abef80e7914..1892ccabb70 100644 --- a/doc/user/project/cycle_analytics.md +++ b/doc/user/project/cycle_analytics.md @@ -3,10 +3,10 @@ > [Introduced][ce-5986] in GitLab 8.12. > > **Note:** -This the first iteration of Cycle Analytics, you can follow the following issue -to track the changes that are coming to this feature: [#20975][ce-20975]. +There are more changes coming to Cycle Analytics, you can follow the following +issue to track the changes to this feature: [#20975][ce-20975]. -Cycle Analytics measures the time it takes to go from [an idea to production] for +Cycle Analytics measures the time it takes to go from an [idea to production] for each project you have. This is achieved by not only indicating the total time it takes to reach at that point, but the total time is broken down into the multiple stages an idea has to pass through to be shipped. @@ -28,9 +28,10 @@ You can see that there are seven stages in total: (first assignment, any milestone, milestone date or assignee is not required) - **Plan** (Board) - Median time from giving an issue a milestone or label until pushing the - first commit + first commit to the branch - **Code** (IDE) - - Median time from the first commit until the merge request is created + - Median time from the first commit to the branch until the merge request is + created - **Test** (CI) - Median total test time for all commits/merges - **Review** (Merge Request/MR) @@ -40,42 +41,42 @@ You can see that there are seven stages in total: - Median time from when the merge request got merged until the deploy to production (production is last stage/environment) - **Production** (Total) - - Sum of all the above stages excluding the Test (CI) time + - Sum of all the above stages' times excluding the Test (CI) time. To clarify, + it's not so much that CI time is "excluded", but rather CI time is already + counted in the review stage since CI is done automatically. Most of the + other stages are purely sequential, but **Test** is not. ## How the data is measured -Cycle Analytics records cycle time so only data on the issues that have been -deployed to production are measured. In case you just started a new project and -you have not pushed anything to production, then you will not be able to -properly see the Cycle Analytics of your project. +Cycle Analytics records cycle time and data based on the project issues with the +exception of the staging and production stages, where only data deployed to +production are measured. Specifically, if your CI is not set up and you have not defined a `production` -[environment], then you will not have any data. +[environment], then you will not have any data for those stages. Below you can see in more detail what the various stages of Cycle Analytics mean. | **Stage** | **Description** | | --------- | --------------- | | Issue | Measures the median time between creating an issue and taking action to solve it, by either labeling it or adding it to a milestone, whatever comes first. The label will be tracked only if it already has an [Issue Board list][board] created for it. | -| Plan | Measures the median time between the action you took for the previous stage, and pushing the first commit to the repository. To make this change tracked, the pushed commit needs to contain the [issue closing pattern], for example `Closes #xxx`, where `xxx` is the number of the issue related to this commit. If the commit does not contain the issue closing pattern, it is not considered to the measurement time of the stage. | -| Code | Measures the median time between pushing a first commit (previous stage) and creating a merge request related to that commit. The key to keep the process tracked is include the [issue closing pattern] to the description of the merge request. | +| Plan | Measures the median time between the action you took for the previous stage, and pushing the first commit to the branch. The very first commit of the branch is the one that triggers the separation between **Plan** and **Code**, and at least one of the commits in the branch needs to contain the related issue number (e.g., `#42`). If none of the commits in the branch mention the related issue number, it is not considered to the measurement time of the stage. | +| Code | Measures the median time between pushing a first commit (previous stage) and creating a merge request (MR) related to that commit. The key to keep the process tracked is to include the [issue closing pattern] to the description of the merge request (for example, `Closes #xxx`, where `xxx` is the number of the issue related to this merge request). If the issue closing pattern is not present in the merge request description, the MR is not considered to the measurement time of the stage. | | Test | Measures the median time to run the entire pipeline for that project. It's related to the time GitLab CI takes to run every job for the commits pushed to that merge request defined in the previous stage. It is basically the start->finish time for all pipelines. `master` is not excluded. It does not attempt to track time for any particular stages. | | Review | Measures the median time taken to review the merge request, between its creation and until it's merged. | -| Staging | Measures the median time between merging the merge request until the very first deployment to production. It's tracked by the [environment] set to `production` in your GitLab CI configuration. If there isn't a `production` environment, this is not tracked. | -| Production| The sum of all time taken to run the entire process, from issue creation to deploying the code to production. | +| Staging | Measures the median time between merging the merge request until the very first deployment to production. It's tracked by the [environment] set to `production` (case-sensitive, `Production` won't work) in your GitLab CI configuration. If there isn't a `production` environment, this is not tracked. | +| Production| The sum of all time (medians) taken to run the entire process, from issue creation to deploying the code to production. | --- Here's a little explanation of how this works behind the scenes: 1. Issues and merge requests are grouped together in pairs, such that for each - `<issue, merge request>` pair, the merge request has `Fixes #xxx` for the - corresponding issue. All other issues and merge requests are **not** considered. - -1. Then the <issue, merge request> pairs are filtered out. Any merge request - that has **not** been deployed to production in the last XX days (specified - by the UI - default is 90 days) prohibits these pairs from being considered. - + `<issue, merge request>` pair, the merge request has the [issue closing pattern] + for the corresponding issue. All other issues and merge requests are **not** + considered. +1. Then the <issue, merge request> pairs are filtered out by last XX days (specified + by the UI - default is 90 days). So it prohibits these pairs from being considered. 1. For the remaining `<issue, merge request>` pairs, we check the information that we need for the stages, like issue creation date, merge request merge time, etc. @@ -83,8 +84,62 @@ Here's a little explanation of how this works behind the scenes: To sum up, anything that doesn't follow the [GitLab flow] won't be tracked at all. So, if a merge request doesn't close an issue or an issue is not labeled with a label present in the Issue Board or assigned a milestone or a project has no -`production` environment, the Cycle Analytics dashboard won't present any data -at all. +`production` environment (for staging and production stages), the Cycle Analytics +dashboard won't present any data at all. + +## Example workflow + +Below is a simple fictional workflow of a single cycle that happens in a +single day passing through all seven stages. Note that if a stage does not have +a start/stop mark, it is not measured and hence not calculated in the median +time. It is assumed that milestones are created and CI for testing and setting +environments is configured. + +1. Issue is created at 09:00 (start of **Issue** stage). +1. Issue is added to a milestone at 11:00 (stop of **Issue** stage / start of + **Plan** stage). +1. Start working on the issue, create a branch locally and make one commit at + 12:00. +1. Make a second commit to the branch which mentions the issue number at 12.30 + (stop of **Plan** stage / start of **Code** stage). +1. Push branch and create a merge request that contains the [issue closing pattern] + in its description at 14:00 (stop of **Code** stage / start of **Test** and + **Review** stages). +1. The CI starts running your scripts defined in [`.gitlab-ci.yml`][yml] and + takes 5min (stop of **Test** stage). +1. Review merge request, ensure that everything is OK and merge the merge + request at 19:00. (stop of **Review** stage / start of **Staging** stage). +1. Now that the merge request is merged, a deployment to the `production` + environment starts and finishes at 19:30 (stop of **Staging** stage). +1. The cycle completes and the sum of the median times of the previous stages + is recorded to the **Production** stage. That is the time between creating an + issue and deploying its relevant merge request to production. + +From the above example you can conclude the time it took each stage to complete +as long as their total time: + +- **Issue**: 2h (11:00 - 09:00) +- **Plan**: 1h (12:00 - 11:00) +- **Code**: 2h (14:00 - 12:00) +- **Test**: 5min +- **Review**: 5h (19:00 - 14:00) +- **Staging**: 30min (19:30 - 19:00) +- **Production**: Since this stage measures the sum of median time off all + previous stages, we cannot calculate it if we don't know the status of the + stages before. In case this is the very first cycle that is run in the project, + then the **Production** time is 10h 30min (19:30 - 09:00) + +A few notes: + +- In the above example we demonstrated that it doesn't matter if your first + commit doesn't mention the issue number, you can do this later in any commit + of the branch you are working on. +- You can see that the **Test** stage is not calculated to the overall time of + the cycle since it is included in the **Review** process (every MR should be + tested). +- The example above was just **one cycle** of the seven stages. Add multiple + cycles, calculate their median time and the result is what the dashboard of + Cycle Analytics is showing. ## Permissions @@ -104,11 +159,12 @@ Learn more about Cycle Analytics in the following resources: - [Cycle Analytics feature highlight](https://about.gitlab.com/2016/09/21/cycle-analytics-feature-highlight/) +[board]: issue_board.md#creating-a-new-list [ce-5986]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5986 [ce-20975]: https://gitlab.com/gitlab-org/gitlab-ce/issues/20975 -[GitLab flow]: ../../workflow/gitlab_flow.md -[permissions]: ../permissions.md [environment]: ../../ci/yaml/README.md#environment -[board]: issue_board.md#creating-a-new-list +[GitLab flow]: ../../workflow/gitlab_flow.md [idea to production]: https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#from-idea-to-production-with-gitlab [issue closing pattern]: issues/automatic_issue_closing.md +[permissions]: ../permissions.md +[yml]: ../../ci/yaml/README.md diff --git a/doc/user/project/git_attributes.md b/doc/user/project/git_attributes.md new file mode 100644 index 00000000000..21ef94e61f7 --- /dev/null +++ b/doc/user/project/git_attributes.md @@ -0,0 +1,22 @@ +# Git Attributes + +GitLab supports defining custom [Git attributes][gitattributes] such as what +files to treat as binary, and what language to use for syntax highlighting +diffs. + +To define these attributes, create a file called `.gitattributes` in the root +directory of your repository and push it to the default branch of your project. + +## Encoding Requirements + +The `.gitattributes` file _must_ be encoded in UTF-8 and _must not_ contain a +Byte Order Mark. If a different encoding is used, the file's contents will be +ignored. + +## Syntax Highlighting + +The `.gitattributes` file can be used to define which language to use when +syntax highlighting files and diffs. See ["Syntax +Highlighting"](highlighting.md) for more information. + +[gitattributes]: https://git-scm.com/docs/gitattributes diff --git a/doc/user/project/img/container_registry_enable.png b/doc/user/project/img/container_registry_enable.png Binary files differnew file mode 100644 index 00000000000..6fffa2a91d8 --- /dev/null +++ b/doc/user/project/img/container_registry_enable.png diff --git a/doc/user/project/img/container_registry_panel.png b/doc/user/project/img/container_registry_panel.png Binary files differnew file mode 100644 index 00000000000..60fd76192b7 --- /dev/null +++ b/doc/user/project/img/container_registry_panel.png diff --git a/doc/user/project/img/container_registry_tab.png b/doc/user/project/img/container_registry_tab.png Binary files differnew file mode 100644 index 00000000000..36b883aaa97 --- /dev/null +++ b/doc/user/project/img/container_registry_tab.png diff --git a/doc/user/project/img/cycle_analytics_landing_page.png b/doc/user/project/img/cycle_analytics_landing_page.png Binary files differindex 4fa42c87395..b212134d5ed 100644 --- a/doc/user/project/img/cycle_analytics_landing_page.png +++ b/doc/user/project/img/cycle_analytics_landing_page.png diff --git a/doc/container_registry/img/mitmproxy-docker.png b/doc/user/project/img/mitmproxy-docker.png Binary files differindex 4e3e37b413d..4e3e37b413d 100644 --- a/doc/container_registry/img/mitmproxy-docker.png +++ b/doc/user/project/img/mitmproxy-docker.png diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index cac926b3e28..4a6c0d88241 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -31,9 +31,10 @@ Below is a table of the definitions used for GitLab's Issue Board. There are three types of lists, the ones you create based on your labels, and two default: -- **Backlog** (default): shows all opened issues that do not fall in one of the other lists. Always appears on the very left. -- **Done** (default): shows all closed issues that do not fall in one of the other lists. Always appears on the very right. -- Label list: a list based on a label. It shows all opened or closed issues with that label. +- **Backlog** (default): shows all issues that do not fall in one of the other lists. Always appears on the very left. +- **Done** (default): shows all closed issues. Always appears on the very right. +Label list: a list based on a label. It shows all issues with that label. +- Label list: a list based on a label. It shows all opened issues with that label.  diff --git a/doc/user/project/merge_requests/img/versions-compare.png b/doc/user/project/merge_requests/img/versions_compare.png Binary files differindex 890cae7768c..890cae7768c 100644 --- a/doc/user/project/merge_requests/img/versions-compare.png +++ b/doc/user/project/merge_requests/img/versions_compare.png diff --git a/doc/user/project/merge_requests/img/versions-dropdown.png b/doc/user/project/merge_requests/img/versions_dropdown.png Binary files differindex 9bab9304e14..9bab9304e14 100644 --- a/doc/user/project/merge_requests/img/versions-dropdown.png +++ b/doc/user/project/merge_requests/img/versions_dropdown.png diff --git a/doc/user/project/merge_requests/img/versions_system_note.png b/doc/user/project/merge_requests/img/versions_system_note.png Binary files differnew file mode 100644 index 00000000000..7c9d7715745 --- /dev/null +++ b/doc/user/project/merge_requests/img/versions_system_note.png diff --git a/doc/user/project/merge_requests/merge_when_build_succeeds.md b/doc/user/project/merge_requests/merge_when_build_succeeds.md index 011f9cbc381..c138061fd40 100644 --- a/doc/user/project/merge_requests/merge_when_build_succeeds.md +++ b/doc/user/project/merge_requests/merge_when_build_succeeds.md @@ -1,16 +1,16 @@ # Merge When Build Succeeds When reviewing a merge request that looks ready to merge but still has one or -more CI builds running, you can set it to be merged automatically when all -builds succeed. This way, you don't have to wait for the builds to finish and -remember to merge the request manually. +more CI builds running, you can set it to be merged automatically when the +builds pipeline succeed. This way, you don't have to wait for the builds to +finish and remember to merge the request manually.  When you hit the "Merge When Build Succeeds" button, the status of the merge request will be updated to represent the impending merge. If you cannot wait -for the build to succeed and want to merge immediately, this option is available -in the dropdown menu on the right of the main button. +for the pipeline to succeed and want to merge immediately, this option is +available in the dropdown menu on the right of the main button. Both team developers and the author of the merge request have the option to cancel the automatic merge if they find a reason why it shouldn't be merged @@ -18,9 +18,9 @@ after all.  -When the build succeeds, the merge request will automatically be merged. When -the build fails, the author gets a chance to retry any failed builds, or to -push new commits to fix the failure. +When the pipeline succeeds, the merge request will automatically be merged. +When the pipeline fails, the author gets a chance to retry any failed builds, +or to push new commits to fix the failure. When the builds are retried and succeed on the second try, the merge request will automatically be merged after all. When the merge request is updated with @@ -40,7 +40,7 @@ hit **Save** for the changes to take effect.  -From now on, every time the build fails you will not be able to merge the merge -request from the UI, until you make the build pass. +From now on, every time the pipelinefails you will not be able to merge the +merge request from the UI, until you make all relevant builds pass.  diff --git a/doc/user/project/merge_requests/versions.md b/doc/user/project/merge_requests/versions.md index 2805fdf635c..77eab7ba5e3 100644 --- a/doc/user/project/merge_requests/versions.md +++ b/doc/user/project/merge_requests/versions.md @@ -7,26 +7,36 @@ of merge request diff is created. When you visit a merge request that contains more than one pushes, you can select and compare the versions of those merge request diffs. - + + +--- By default, the latest version of changes is shown. However, you can select an older one from version dropdown. - + + +--- -You can also compare the merge request version with older one to see what is +You can also compare the merge request version with an older one to see what has changed since then. - + + +--- + +Every time you push new changes to the branch, a link to compare the last +changes appears as a system note. -Please note that comments are disabled while viewing outdated merge versions -or comparing to versions other than base. + --- ->**Note:** -Merge request versions are based on push not on commit. So, if you pushed 5 -commits in a single push, it will be a single option in the dropdown. If you -pushed 5 times, that will count for 5 options. +>**Notes:** +- Comments are disabled while viewing outdated merge versions or comparing to + versions other than base. +- Merge request versions are based on push not on commit. So, if you pushed 5 + commits in a single push, it will be a single option in the dropdown. If you + pushed 5 times, that will count for 5 options. [ce-5467]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5467 diff --git a/doc/user/project/new_ci_build_permissions_model.md b/doc/user/project/new_ci_build_permissions_model.md index e73f60023b5..8827b501901 100644 --- a/doc/user/project/new_ci_build_permissions_model.md +++ b/doc/user/project/new_ci_build_permissions_model.md @@ -87,17 +87,6 @@ your Runners in the most possible secure way, by avoiding the following: By using an insecure GitLab Runner configuration, you allow the rogue developers to steal the tokens of other builds. -## Debugging problems - -With the new permission model in place, there may be times that your build will -fail. This is most likely because your project tries to access other project's -sources, and you don't have the appropriate permissions. In the build log look -for information about 403 or forbidden access messages - -As an Administrator, you can verify that the user is a member of the group or -project they're trying to have access to, and you can impersonate the user to -retry the failing build in order to verify that everything is correct. - ## Build triggers [Build triggers][triggers] do not support the new permission model. @@ -149,17 +138,46 @@ with GitLab 8.12. ## Making use of the new CI build permissions model -With the new build permission model, there is now an easy way to access all +With the new build permissions model, there is now an easy way to access all dependent source code in a project. That way, we can: 1. Access a project's Git submodules 1. Access private container images 1. Access project's and submodule LFS objects -Let's see how that works with Git submodules and private Docker images hosted on +Below you can see the prerequisites needed to make use of the new permissions +model and how that works with Git submodules and private Docker images hosted on the container registry. -## Git submodules +### Prerequisites to use the new permissions model + +With the new permissions model in place, there may be times that your build will +fail. This is most likely because your project tries to access other project's +sources, and you don't have the appropriate permissions. In the build log look +for information about 403 or forbidden access messages. + +In short here's what you need to do should you encounter any issues. + +As an administrator: + +- **500 errors**: You will need to update [GitLab Workhorse][workhorse] to at + least 0.8.2. This is done automatically for Omnibus installations, you need to + [check manually][update-docs] for installations from source. +- **500 errors**: Check if you have another web proxy sitting in front of NGINX (HAProxy, + Apache, etc.). It might be a good idea to let GitLab use the internal NGINX + web server and not disable it completely. See [this comment][comment] for an + example. +- **403 errors**: You need to make sure that your installation has [HTTP(S) + cloning enabled][https]. HTTP(S) support is now a **requirement** by GitLab CI + to clone all sources. + +As a user: + +- Make sure you are a member of the group or project you're trying to have + access to. As an Administrator, you can verify that by impersonating the user + and retry the failing build in order to verify that everything is correct. + +### Git submodules > It often happens that while working on one project, you need to use another @@ -283,7 +301,11 @@ test: - docker run $CI_REGISTRY/group/other-project:latest ``` -[git-scm]: https://git-scm.com/book/en/v2/Git-Tools-Submodules [build permissions]: ../permissions.md#builds-permissions +[comment]: https://gitlab.com/gitlab-org/gitlab-ce/issues/22484#note_16648302 [ext]: ../permissions.md#external-users +[git-scm]: https://git-scm.com/book/en/v2/Git-Tools-Submodules +[https]: ../admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols [triggers]: ../../ci/triggers/README.md +[update-docs]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update +[workhorse]: https://gitlab.com/gitlab-org/gitlab-workhorse diff --git a/doc/user/project/repository/img/web_editor_new_branch_from_issue.png b/doc/user/project/repository/img/web_editor_new_branch_from_issue.png Binary files differnew file mode 100644 index 00000000000..b0a63ddf0ab --- /dev/null +++ b/doc/user/project/repository/img/web_editor_new_branch_from_issue.png diff --git a/doc/user/project/repository/web_editor.md b/doc/user/project/repository/web_editor.md index 993c6bfb7e9..675e89e4247 100644 --- a/doc/user/project/repository/web_editor.md +++ b/doc/user/project/repository/web_editor.md @@ -97,11 +97,11 @@ There are multiple ways to create a branch from GitLab's web interface. In case your development workflow dictates to have an issue for every merge request, you can quickly create a branch right on the issue page which will be -tied with the issue itself. You can see a **New Branch** button after the issue +tied with the issue itself. You can see a **New branch** button after the issue description, unless there is already a branch with the same name or a referenced merge request. - + Once you click it, a new branch will be created that diverges from the default branch of your project, by default `master`. The branch name will be based on diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index 445c0ee8333..dfc762fe1d3 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -7,7 +7,8 @@ > that of the exporter. > - For existing installations, the project import option has to be enabled in > application settings (`/admin/application_settings`) under 'Import sources'. -> You will have to be an administrator to enable and use the import functionality. +> Ask your administrator if you don't see the **GitLab export** button when +> creating a new project. > - You can find some useful raketasks if you are an administrator in the > [import_export](../../../administration/raketasks/project_import_export.md) > raketask. @@ -21,7 +22,8 @@ with all their related data and be moved into a new GitLab instance. | GitLab version | Import/Export version | | -------- | -------- | -| 8.12.0 to current | 0.1.4 | +| 8.13.0 to current | 0.1.5 | +| 8.12.0 | 0.1.4 | | 8.10.3 | 0.1.3 | | 8.10.0 | 0.1.2 | | 8.9.5 | 0.1.1 | diff --git a/doc/user/project/slash_commands.md b/doc/user/project/slash_commands.md index 1792a0c501d..5f6a6c6503e 100644 --- a/doc/user/project/slash_commands.md +++ b/doc/user/project/slash_commands.md @@ -27,4 +27,5 @@ do. | `/subscribe` | Subscribe | | `/unsubscribe` | Unsubscribe | | <code>/due <in 2 days | this Friday | December 31st></code> | Set due date | -| `/remove_due_date` | Remove due date | +| `/remove_due_date` | Remove due date | +| `/wip` | Toggle the Work In Progress status | diff --git a/doc/workflow/gitlab_flow.md b/doc/workflow/gitlab_flow.md index 7c0eb90d540..2215f37b81a 100644 --- a/doc/workflow/gitlab_flow.md +++ b/doc/workflow/gitlab_flow.md @@ -228,7 +228,7 @@ We'll discuss the three reasons to merge in master: leveraging code, merge confl If you need to leverage some code that was introduced in master after you created the feature branch you can sometimes solve this by just cherry-picking a commit. If your feature branch has a merge conflict, creating a merge commit is a normal way of solving this. You can prevent some merge conflicts by using [gitattributes](http://git-scm.com/docs/gitattributes) for files that can be in a random order. -For example in GitLab our changelog file is specified in .gitattributes as `CHANGELOG merge=union` so that there are fewer merge conflicts in it. +For example in GitLab our changelog file is specified in .gitattributes as `CHANGELOG.md merge=union` so that there are fewer merge conflicts in it. The last reason for creating merge commits is having long lived branches that you want to keep up to date with the latest state of the project. Martin Fowler, in [his article about feature branches](http://martinfowler.com/bliki/FeatureBranch.html) talks about this Continuous Integration (CI). At GitLab we are guilty of confusing CI with branch testing. Quoting Martin Fowler: "I've heard people say they are doing CI because they are running builds, perhaps using a CI server, on every branch with every commit. diff --git a/doc/workflow/importing/migrating_from_svn.md b/doc/workflow/importing/migrating_from_svn.md index 4828bb5dce6..423b095e69e 100644 --- a/doc/workflow/importing/migrating_from_svn.md +++ b/doc/workflow/importing/migrating_from_svn.md @@ -4,6 +4,112 @@ Subversion (SVN) is a central version control system (VCS) while Git is a distributed version control system. There are some major differences between the two, for more information consult your favorite search engine. +## Overview + +There are two approaches to SVN to Git migration: + +1. [Git/SVN Mirror](#smooth-migration-with-a-gitsvn-mirror-using-subgit) which: + - Makes the GitLab repository to mirror the SVN project. + - Git and SVN repositories are kept in sync; you can use either one. + - Smoothens the migration process and allows to manage migration risks. + +1. [Cut over migration](#cut-over-migration-with-svn2git) which: + - Translates and imports the existing data and history from SVN to Git. + - Is a fire and forget approach, good for smaller teams. + +## Smooth migration with a Git/SVN mirror using SubGit + +[SubGit](https://subgit.com) is a tool for a smooth, stress-free SVN to Git +migration. It creates a writable Git mirror of a local or remote Subversion +repository and that way you can use both Subversion and Git as long as you like. +It requires access to your GitLab server as it talks with the Git repositories +directly in a filesystem level. + +### SubGit prerequisites + +1. Install Oracle JRE 1.8 or newer. On Debian-based Linux distributions you can + follow [this article](http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html). +1. Download SubGit from https://subgit.com/download/. +1. Unpack the downloaded SubGit zip archive to the `/opt` directory. The `subgit` + command will be available at `/opt/subgit-VERSION/bin/subgit`. + +### SubGit configuration + +The first step to mirror you SVN repository in GitLab is to create a new empty +project which will be used as a mirror. For Omnibus installations the path to +the repository will be located at +`/var/opt/gitlab/git-data/repositories/USER/REPO.git` by default. For +installations from source, the default repository directory will be +`/home/git/repositories/USER/REPO.git`. For convenience, assign this path to a +variable: + +``` +GIT_REPO_PATH=/var/opt/gitlab/git-data/repositories/USER/REPOS.git +``` + +SubGit will keep this repository in sync with a remote SVN project. For +convenience, assign your remote SVN project URL to a variable: + +``` +SVN_PROJECT_URL=http://svn.company.com/repos/project +``` + +Next you need to run SubGit to set up a Git/SVN mirror. Make sure the following +`subgit` command is ran on behalf of the same user that keeps ownership of +GitLab Git repositories (by default `git`): + +``` +subgit configure --layout auto $SVN_PROJECT_URL $GIT_REPO_PATH +``` + +Adjust authors and branches mappings, if necessary. Open with your favorite +text editor: + +``` +edit $GIT_REPO_PATH/subgit/authors.txt +edit $GIT_REPO_PATH/subgit/config +``` + +For more information regarding the SubGit configuration options, refer to +[SubGit's documentation](https://subgit.com/documentation.html) website. + +### Initial translation + +Now that SubGit has configured the Git/SVN repos, run `subgit` to perform the +initial translation of existing SVN revisions into the Git repository: + +``` +subgit install $GIT_REPOS_PATH +``` + +After the initial translation is completed, the Git repository and the SVN +project will be kept in sync by `subgit` - new Git commits will be translated to +SVN revisions and new SVN revisions will be translated to Git commits. Mirror +works transparently and does not require any special commands. + +If you would prefer to perform one-time cut over migration with `subgit`, use +the `import` command instead of `install`: + +``` +subgit import $GIT_REPO_PATH +``` + +### SubGit licensing + +Running SubGit in a mirror mode requires a +[registration](https://subgit.com/pricing.html). Registration is free for open +source, academic and startup projects. + +We're currently working on deeper GitLab/SubGit integration. You may track our +progress at [this issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/990). + +### SubGit support + +For any questions related to SVN to GitLab migration with SubGit, you can +contact the SubGit team directly at [support@subgit.com](mailto:support@subgit.com). + +## Cut over migration with svn2git + If you are currently using an SVN repository, you can migrate the repository to Git and GitLab. We recommend a hard cut over - run the migration command once and then have all developers start using the new GitLab repository immediately. @@ -75,5 +181,3 @@ git push --tags origin ## Contribute to this guide We welcome all contributions that would expand this guide with instructions on how to migrate from SVN and other version control systems. - - |
