diff options
| author | karen Carias <karen@gitlab.com> | 2015-09-30 13:33:51 -0700 |
|---|---|---|
| committer | karen Carias <karen@gitlab.com> | 2015-09-30 13:33:51 -0700 |
| commit | 126dd008e110dce5dfe7d1d076bc57f33250ee7b (patch) | |
| tree | 4ce96f2fd7e50faff65df7b556153c08d16a888c /doc | |
| parent | 6a1d695f861e4c5251a2333c673f78705b34891f (diff) | |
| parent | 54452412f765d9e6e6166e105db9adbc7553aec2 (diff) | |
| download | gitlab-ce-126dd008e110dce5dfe7d1d076bc57f33250ee7b.tar.gz | |
solved conflict
Diffstat (limited to 'doc')
62 files changed, 2984 insertions, 273 deletions
diff --git a/doc/README.md b/doc/README.md index 337c4e6a62d..a0ff856ebb6 100644 --- a/doc/README.md +++ b/doc/README.md @@ -15,6 +15,23 @@ - [Web hooks](web_hooks/web_hooks.md) Let GitLab notify you when new code has been pushed to your project. - [Workflow](workflow/README.md) Using GitLab functionality and importing projects from GitHub and SVN. +## CI Documentation + ++ [Quick Start](ci/quick_start/README.md) ++ [Configuring project (.gitlab-ci.yml)](ci/yaml/README.md) ++ [Configuring runner](ci/runners/README.md) ++ [Configuring deployment](ci/deployment/README.md) ++ [Using Docker Images](ci/docker/using_docker_images.md) ++ [Using Docker Build](ci/docker/using_docker_build.md) ++ [Using Variables](ci/variables/README.md) + +### CI Examples + ++ [Test and deploy Ruby applications to Heroku](ci/examples/test-and-deploy-ruby-application-to-heroku.md) ++ [Test and deploy Python applications to Heroku](ci/examples/test-and-deploy-python-application-to-heroku.md) ++ [Test Clojure applications](ci/examples/test-clojure-application.md) ++ Help your favorite programming language and GitLab by sending a merge request with a guide for that language. + ## Administrator documentation - [Custom git hooks](hooks/custom_hooks.md) Custom git hooks (on the filesystem) for when web hooks aren't enough. @@ -29,7 +46,13 @@ - [System hooks](system_hooks/system_hooks.md) Notifications when users, projects and keys are changed. - [Update](update/README.md) Update guides to upgrade your installation. - [Welcome message](customization/welcome_message.md) Add a custom welcome message to the sign-in page. -- [Reply by email](reply_by_email/README.md) Allow users to comment on issues and merge requests by replying to notification emails. +- [Reply by email](incoming_email/README.md) Allow users to comment on issues and merge requests by replying to notification emails. +- [Migrate GitLab CI to CE/EE](migrate_ci_to_ce/README.md) Follow this guide to migrate your existing GitLab CI data to GitLab CE/EE. + +### Administrator documentation + ++ [User permissions](permissions/permissions.md) ++ [API](api/README.md) ## Contributor documentation diff --git a/doc/api/projects.md b/doc/api/projects.md index 10533c73a31..96485857035 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -515,6 +515,8 @@ Parameters: "push_events": "true", "issues_events": "true", "merge_requests_events": "true", + "note_events": "true", + "enable_ssl_verification": "true", "created_at": "2012-10-12T17:04:47Z" } ``` @@ -535,6 +537,8 @@ Parameters: - `issues_events` - Trigger hook on issues events - `merge_requests_events` - Trigger hook on merge_requests events - `tag_push_events` - Trigger hook on push_tag events +- `note_events` - Trigger hook on note events +- `enable_ssl_verification` - Do SSL verification when triggering the hook ### Edit project hook @@ -553,6 +557,8 @@ Parameters: - `issues_events` - Trigger hook on issues events - `merge_requests_events` - Trigger hook on merge_requests events - `tag_push_events` - Trigger hook on push_tag events +- `note_events` - Trigger hook on note events +- `enable_ssl_verification` - Do SSL verification when triggering the hook ### Delete project hook diff --git a/doc/api/services.md b/doc/api/services.md index bc5049dd302..7d45b2cf463 100644 --- a/doc/api/services.md +++ b/doc/api/services.md @@ -27,6 +27,14 @@ Delete Asana service for a project. DELETE /projects/:id/services/asana ``` +### Get Asana service settings + +Get Asana service settings for a project. + +``` +GET /projects/:id/services/asana +``` + ## Assembla Project Management Software (Source Commits Endpoint) @@ -52,6 +60,14 @@ Delete Assembla service for a project. DELETE /projects/:id/services/assembla ``` +### Get Assembla service settings + +Get Assembla service settings for a project. + +``` +GET /projects/:id/services/assembla +``` + ## Atlassian Bamboo CI A continuous integration and build server @@ -81,6 +97,14 @@ Delete Atlassian Bamboo CI service for a project. DELETE /projects/:id/services/bamboo ``` +### Get Atlassian Bamboo CI service settings + +Get Atlassian Bamboo CI service settings for a project. + +``` +GET /projects/:id/services/bamboo +``` + ## Buildkite Continuous integration and deployments @@ -107,6 +131,14 @@ Delete Buildkite service for a project. DELETE /projects/:id/services/buildkite ``` +### Get Buildkite service settings + +Get Buildkite service settings for a project. + +``` +GET /projects/:id/services/buildkite +``` + ## Campfire Simple web-based real-time group chat @@ -133,6 +165,14 @@ Delete Campfire service for a project. DELETE /projects/:id/services/campfire ``` +### Get Campfire service settings + +Get Campfire service settings for a project. + +``` +GET /projects/:id/services/campfire +``` + ## Custom Issue Tracker Custom issue tracker @@ -161,6 +201,14 @@ Delete Custom Issue Tracker service for a project. DELETE /projects/:id/services/custom-issue-tracker ``` +### Get Custom Issue Tracker service settings + +Get Custom Issue Tracker service settings for a project. + +``` +GET /projects/:id/services/custom-issue-tracker +``` + ## Drone CI Drone is a Continuous Integration platform built on Docker, written in Go @@ -187,6 +235,14 @@ Delete Drone CI service for a project. DELETE /projects/:id/services/drone-ci ``` +### Get Drone CI service settings + +Get Drone CI service settings for a project. + +``` +GET /projects/:id/services/drone-ci +``` + ## Emails on push Email the commits and diff of each push to a list of recipients. @@ -213,6 +269,14 @@ Delete Emails on push service for a project. DELETE /projects/:id/services/emails-on-push ``` +### Get Emails on push service settings + +Get Emails on push service settings for a project. + +``` +GET /projects/:id/services/emails-on-push +``` + ## External Wiki Replaces the link to the internal wiki with a link to an external wiki. @@ -237,6 +301,14 @@ Delete External Wiki service for a project. DELETE /projects/:id/services/external-wiki ``` +### Get External Wiki service settings + +Get External Wiki service settings for a project. + +``` +GET /projects/:id/services/external-wiki +``` + ## Flowdock Flowdock is a collaboration web app for technical teams. @@ -261,6 +333,14 @@ Delete Flowdock service for a project. DELETE /projects/:id/services/flowdock ``` +### Get Flowdock service settings + +Get Flowdock service settings for a project. + +``` +GET /projects/:id/services/flowdock +``` + ## Gemnasium Gemnasium monitors your project dependencies and alerts you about updates and security vulnerabilities. @@ -286,6 +366,14 @@ Delete Gemnasium service for a project. DELETE /projects/:id/services/gemnasium ``` +### Get Gemnasium service settings + +Get Gemnasium service settings for a project. + +``` +GET /projects/:id/services/gemnasium +``` + ## GitLab CI Continuous integration server from GitLab @@ -312,6 +400,14 @@ Delete GitLab CI service for a project. DELETE /projects/:id/services/gitlab-ci ``` +### Get GitLab CI service settings + +Get GitLab CI service settings for a project. + +``` +GET /projects/:id/services/gitlab-ci +``` + ## HipChat Private group chat and IM @@ -341,6 +437,14 @@ Delete HipChat service for a project. DELETE /projects/:id/services/hipchat ``` +### Get HipChat service settings + +Get HipChat service settings for a project. + +``` +GET /projects/:id/services/hipchat +``` + ## Irker (IRC gateway) Send IRC messages, on update, to a list of recipients through an Irker gateway. @@ -371,6 +475,14 @@ Delete Irker (IRC gateway) service for a project. DELETE /projects/:id/services/irker ``` +### Get Irker (IRC gateway) service settings + +Get Irker (IRC gateway) service settings for a project. + +``` +GET /projects/:id/services/irker +``` + ## JIRA Jira issue tracker @@ -400,6 +512,14 @@ Delete JIRA service for a project. DELETE /projects/:id/services/jira ``` +### Get JIRA service settings + +Get JIRA service settings for a project. + +``` +GET /projects/:id/services/jira +``` + ## PivotalTracker Project Management Software (Source Commits Endpoint) @@ -424,6 +544,14 @@ Delete PivotalTracker service for a project. DELETE /projects/:id/services/pivotaltracker ``` +### Get PivotalTracker service settings + +Get PivotalTracker service settings for a project. + +``` +GET /projects/:id/services/pivotaltracker +``` + ## Pushover Pushover makes it easy to get real-time notifications on your Android device, iPhone, iPad, and Desktop. @@ -452,6 +580,14 @@ Delete Pushover service for a project. DELETE /projects/:id/services/pushover ``` +### Get Pushover service settings + +Get Pushover service settings for a project. + +``` +GET /projects/:id/services/pushover +``` + ## Redmine Redmine issue tracker @@ -479,6 +615,14 @@ Delete Redmine service for a project. DELETE /projects/:id/services/redmine ``` +### Get Redmine service settings + +Get Redmine service settings for a project. + +``` +GET /projects/:id/services/redmine +``` + ## Slack A team communication tool for the 21st century @@ -505,6 +649,14 @@ Delete Slack service for a project. DELETE /projects/:id/services/slack ``` +### Get Slack service settings + +Get Slack service settings for a project. + +``` +GET /projects/:id/services/slack +``` + ## JetBrains TeamCity CI A continuous integration and build server @@ -534,3 +686,11 @@ Delete JetBrains TeamCity CI service for a project. DELETE /projects/:id/services/teamcity ``` +### Get JetBrains TeamCity CI service settings + +Get JetBrains TeamCity CI service settings for a project. + +``` +GET /projects/:id/services/teamcity +``` + diff --git a/doc/ci/README.md b/doc/ci/README.md new file mode 100644 index 00000000000..97325069ceb --- /dev/null +++ b/doc/ci/README.md @@ -0,0 +1,23 @@ +## GitLab CI Documentation + +### User documentation + ++ [Quick Start](quick_start/README.md) ++ [Configuring project (.gitlab-ci.yml)](yaml/README.md) ++ [Configuring runner](runners/README.md) ++ [Configuring deployment](deployment/README.md) ++ [Using Docker Images](docker/using_docker_images.md) ++ [Using Docker Build](docker/using_docker_build.md) ++ [Using Variables](variables/README.md) + +### Examples + ++ [Test and deploy Ruby applications to Heroku](examples/test-and-deploy-ruby-application-to-heroku.md) ++ [Test and deploy Python applications to Heroku](examples/test-and-deploy-python-application-to-heroku.md) ++ [Test Clojure applications](examples/test-clojure-application.md) ++ Help your favorite programming language and GitLab by sending a merge request with a guide for that language. + +### Administrator documentation + ++ [User permissions](permissions/README.md) ++ [API](api/README.md) diff --git a/doc/ci/api/README.md b/doc/ci/api/README.md new file mode 100644 index 00000000000..33c5b172e98 --- /dev/null +++ b/doc/ci/api/README.md @@ -0,0 +1,86 @@ +# GitLab CI API + +## Resources + +- [Projects](projects.md) +- [Runners](runners.md) +- [Commits](commits.md) +- [Builds](builds.md) + + +## Authentication + +GitLab CI API uses different types of authentication depends on what API you use. +Each API document has section with information about authentication you need to use. + +GitLab CI API has 4 authentication methods: + +* GitLab user token & GitLab url +* GitLab CI project token +* GitLab CI runners registration token +* GitLab CI runner token + + +### Authentication #1: GitLab user token & GitLab url + +Authentication is done by +sending the `private-token` of a valid user and the `url` of an +authorized Gitlab instance via a query string along with the API +request: + + GET http://gitlab.example.com/ci/api/v1/projects?private_token=QVy1PB7sTxfy4pqfZM1U&url=http://demo.gitlab.com/ + +If preferred, you may instead send the `private-token` as a header in +your request: + + curl --header "PRIVATE-TOKEN: QVy1PB7sTxfy4pqfZM1U" "http://gitlab.example.com/ci/api/v1/projects?url=http://demo.gitlab.com/" + + +### Authentication #2: GitLab CI project token + +Each project in GitLab CI has it own token. +It can be used to get project commits and builds information. +You can use project token only for certain project. + +### Authentication #3: GitLab CI runners registration token + +This token is not persisted and is generated on each application start. +It can be used only for registering new runners in system. You can find it on +GitLab CI Runners web page https://gitlab-ci.example.com/admin/runners + +### Authentication #4: GitLab CI runner token + +Every GitLab CI runner has it own token that allow it to receive and update +GitLab CI builds. This token exists of internal purposes and should be used only +by runners + +## JSON + +All API requests are serialized using JSON. You don't need to specify +`.json` at the end of API URL. + +## Status codes + +The API is designed to return different status codes according to context and action. In this way if a request results in an error the caller is able to get insight into what went wrong, e.g. status code `400 Bad Request` is returned if a required attribute is missing from the request. The following list gives an overview of how the API functions generally behave. + +API request types: + +- `GET` requests access one or more resources and return the result as JSON +- `POST` requests return `201 Created` if the resource is successfully created and return the newly created resource as JSON +- `GET`, `PUT` and `DELETE` return `200 OK` if the resource is accessed, modified or deleted successfully, the (modified) result is returned as JSON +- `DELETE` requests are designed to be idempotent, meaning a request a resource still returns `200 OK` even it was deleted before or is not available. The reasoning behind it is the user is not really interested if the resource existed before or not. + +The following list shows the possible return codes for API requests. + +Return values: + +- `200 OK` - The `GET`, `PUT` or `DELETE` request was successful, the resource(s) itself is returned as JSON +- `201 Created` - The `POST` request was successful and the resource is returned as JSON +- `400 Bad Request` - A required attribute of the API request is missing, e.g. the title of an issue is not given +- `401 Unauthorized` - The user is not authenticated, a valid user token is necessary, see above +- `403 Forbidden` - The request is not allowed, e.g. the user is not allowed to delete a project +- `404 Not Found` - A resource could not be accessed, e.g. an ID for a resource could not be found +- `405 Method Not Allowed` - The request is not supported +- `409 Conflict` - A conflicting resource already exists, e.g. creating a project with a name that already exists +- `422 Unprocessable` - The entity could not be processed +- `500 Server Error` - While handling the request something went wrong on the server side diff --git a/doc/ci/api/builds.md b/doc/ci/api/builds.md new file mode 100644 index 00000000000..3b5008ccdb4 --- /dev/null +++ b/doc/ci/api/builds.md @@ -0,0 +1,41 @@ +# Builds API + +This API used by runners to receive and update builds. + +__Authentication is done by runner token__ + +## Builds + +### Runs oldest pending build by runner + + POST /ci/builds/register + +Parameters: + + * `token` (required) - The unique token of runner + +Returns: + +```json +{ + "id" : 79, + "commands" : "", + "path" : "", + "ref" : "", + "sha" : "", + "project_id" : 6, + "repo_url" : "git@demo.gitlab.com:gitlab/gitlab-shell.git", + "before_sha" : "" +} +``` + + +### Update details of an existing build + + PUT /ci/builds/:id + +Parameters: + + * `id` (required) - The ID of a project + * `state` (optional) - The state of a build + * `trace` (optional) - The trace of a build diff --git a/doc/ci/api/commits.md b/doc/ci/api/commits.md new file mode 100644 index 00000000000..4df7afc6c52 --- /dev/null +++ b/doc/ci/api/commits.md @@ -0,0 +1,101 @@ +# Commits API + +__Authentication is done by GitLab CI project token__ + +## Commits + +### Retrieve all commits per project + +Get list of commits per project + + GET /ci/commits + +Parameters: + + * `project_id` (required) - The ID of a project + * `project_token` (requires) - Project token + * `page` (optional) + * `per_page` (optional) - items per request (default is 20) + +Returns: + +```json +[{ + "id": 3, + "ref": "master", + "sha": "65617dfc36761baa1f46a7006f2a88916f7f56cf", + "project_id": 2, + "before_sha": "96906f2bceb04c7323f8514aa5ad8cb1313e2898", + "created_at": "2014-11-05T09:46:35.247Z", + "status": "success", + "finished_at": "2014-11-05T09:46:44.254Z", + "duration": 5.062692165374756, + "git_commit_message": "wow\n", + "git_author_name": "Administrator", + "git_author_email": "admin@example.com", + "builds": [{ + "id": 7, + "project_id": 2, + "ref": "master", + "status": "success", + "finished_at": "2014-11-05T09:46:44.254Z", + "created_at": "2014-11-05T09:46:35.259Z", + "updated_at": "2014-11-05T09:46:44.255Z", + "sha": "65617dfc36761baa1f46a7006f2a88916f7f56cf", + "started_at": "2014-11-05T09:46:39.192Z", + "before_sha": "96906f2bceb04c7323f8514aa5ad8cb1313e2898", + "runner_id": 1, + "coverage": null, + "commit_id": 3 + }] +}] +``` + +### Create commit + +Inform GitLab CI about new commit you want it to build. + +__If commit already exists in GitLab CI it will not be created__ + + + POST /ci/commits + +Parameters: + + * `project_id` (required) - The ID of a project + * `project_token` (requires) - Project token + * `data` (required) - Push data. For example see comment in `lib/api/commits.rb` + +Returns: + +```json +{ + "id": 3, + "ref": "master", + "sha": "65617dfc36761baa1f46a7006f2a88916f7f56cf", + "project_id": 2, + "before_sha": "96906f2bceb04c7323f8514aa5ad8cb1313e2898", + "created_at": "2014-11-05T09:46:35.247Z", + "status": "success", + "finished_at": "2014-11-05T09:46:44.254Z", + "duration": 5.062692165374756, + "git_commit_message": "wow\n", + "git_author_name": "Administrator", + "git_author_email": "admin@example.com", + "builds": [{ + "id": 7, + "project_id": 2, + "ref": "master", + "status": "success", + "finished_at": "2014-11-05T09:46:44.254Z", + "created_at": "2014-11-05T09:46:35.259Z", + "updated_at": "2014-11-05T09:46:44.255Z", + "sha": "65617dfc36761baa1f46a7006f2a88916f7f56cf", + "started_at": "2014-11-05T09:46:39.192Z", + "before_sha": "96906f2bceb04c7323f8514aa5ad8cb1313e2898", + "runner_id": 1, + "coverage": null, + "commit_id": 3 + }] +} +``` diff --git a/doc/ci/api/projects.md b/doc/ci/api/projects.md new file mode 100644 index 00000000000..5585191e826 --- /dev/null +++ b/doc/ci/api/projects.md @@ -0,0 +1,149 @@ +# Projects API + +This API is intended to aid in the setup and configuration of +projects on Gitlab CI. + +__Authentication is done by GitLab user token & GitLab url__ + +## Projects + +### List Authorized Projects + +Lists all projects that the authenticated user has access to. + +``` +GET /ci/projects +``` + +Returns: + +```json + [ + { + "id" : 271, + "name" : "gitlabhq", + "timeout" : 1800, + "token" : "iPWx6WM4lhHNedGfBpPJNP", + "default_ref" : "master", + "gitlab_url" : "http://demo.gitlabhq.com/gitlab/gitlab-shell", + "path" : "gitlab/gitlab-shell", + "always_build" : false, + "polling_interval" : null, + "public" : false, + "ssh_url_to_repo" : "git@demo.gitlab.com:gitlab/gitlab-shell.git", + "gitlab_id" : 3 + }, + { + "id" : 272, + "name" : "gitlab-ci", + "timeout" : 1800, + "token" : "iPWx6WM4lhHNedGfBpPJNP", + "default_ref" : "master", + "gitlab_url" : "http://demo.gitlabhq.com/gitlab/gitlab-shell", + "path" : "gitlab/gitlab-shell", + "always_build" : false, + "polling_interval" : null, + "public" : false, + "ssh_url_to_repo" : "git@demo.gitlab.com:gitlab/gitlab-shell.git", + "gitlab_id" : 4 + } +] +``` + +### List Owned Projects + +Lists all projects that the authenticated user owns. + +``` +GET /ci/projects/owned +``` + +Returns: + +```json +[ + { + "id" : 272, + "name" : "gitlab-ci", + "timeout" : 1800, + "token" : "iPWx6WM4lhHNedGfBpPJNP", + "default_ref" : "master", + "gitlab_url" : "http://demo.gitlabhq.com/gitlab/gitlab-shell", + "path" : "gitlab/gitlab-shell", + "always_build" : false, + "polling_interval" : null, + "public" : false, + "ssh_url_to_repo" : "git@demo.gitlab.com:gitlab/gitlab-shell.git", + "gitlab_id" : 4 + } +] +``` + +### Single Project + +Returns information about a single project for which the user is +authorized. + + GET /ci/projects/:id + +Parameters: + + * `id` (required) - The ID of the Gitlab CI project + +### Create Project + +Creates a Gitlab CI project using Gitlab project details. + + POST /ci/projects + +Parameters: + + * `name` (required) - The name of the project + * `gitlab_id` (required) - The ID of the project on the Gitlab instance + * `default_ref` (optional) - The branch to run on (default to `master`) + +### Update Project + +Updates a Gitlab CI project using Gitlab project details that the +authenticated user has access to. + + PUT /ci/projects/:id + +Parameters: + + * `name` - The name of the project + * `default_ref` - The branch to run on (default to `master`) + +### Remove Project + +Removes a Gitlab CI project that the authenticated user has access to. + + DELETE /ci/projects/:id + +Parameters: + + * `id` (required) - The ID of the Gitlab CI project + +### Link Project to Runner + +Links a runner to a project so that it can make builds (only via +authorized user). + + POST /ci/projects/:id/runners/:runner_id + +Parameters: + + * `id` (required) - The ID of the Gitlab CI project + * `runner_id` (required) - The ID of the Gitlab CI runner + +### Remove Project from Runner + +Removes a runner from a project so that it can not make builds (only +via authorized user). + + DELETE /ci/projects/:id/runners/:runner_id + +Parameters: + + * `id` (required) - The ID of the Gitlab CI project + * `runner_id` (required) - The ID of the Gitlab CI runner
\ No newline at end of file diff --git a/doc/ci/api/runners.md b/doc/ci/api/runners.md new file mode 100644 index 00000000000..e9f88ee066e --- /dev/null +++ b/doc/ci/api/runners.md @@ -0,0 +1,77 @@ +# Runners API + +## Runners + +### Retrieve all runners + +__Authentication is done by GitLab user token & GitLab url__ + +Used to get information about all runners registered on the Gitlab CI +instance. + + GET /ci/runners + +Returns: + +```json +[ + { + "id" : 85, + "token" : "12b68e90394084703135" + }, + { + "id" : 86, + "token" : "76bf894e969364709864" + }, +] +``` + +### Register a new runner + + +__Authentication is done with a Shared runner registration token or a project Specific runner registration token__ + +Used to make Gitlab CI aware of available runners. + + POST /ci/runners/register + +Parameters: + + * `token` (required) - The registration token. It is 2 types of token you can pass here. + +1. Shared runner registration token +2. Project specific registration token + +Returns: + +```json +{ + "id" : 85, + "token" : "12b68e90394084703135" +} +``` + +### Delete a runner + + +__Authentication is done by runner token__ + +Used to removing runners. + + DELETE /ci/runners/delete + +Parameters: + + * `token` (required) - The runner token. + +Returns: + +```json +{ + "id" : 1, + "token" : "d14963981a428f70121777e50643d1", + "created_at" : "2015-02-26T11:39:39.232Z", + "updated_at" : "2015-02-26T11:39:39.232Z", + "description" : "awesome runner" +} +```
\ No newline at end of file diff --git a/doc/ci/deployment/README.md b/doc/ci/deployment/README.md new file mode 100644 index 00000000000..ffd841ca9e7 --- /dev/null +++ b/doc/ci/deployment/README.md @@ -0,0 +1,98 @@ +## Using Dpl as deployment tool +Dpl (dee-pee-ell) is a deploy tool made for continuous deployment that's developed and used by Travis CI, but can also be used with GitLab CI. + +**We recommend to use Dpl, if you're deploying to any of these of these services: https://github.com/travis-ci/dpl#supported-providers**. + +### Requirements +To use Dpl you need at least Ruby 1.8.7 with ability to install gems. + +### Basic usage +The Dpl can be installed on any machine with: +``` +gem install dpl +``` + +This allows you to test all commands from your shell, rather than having to test it on a CI server. + +If you don't have Ruby installed you can do it on Debian-compatible Linux with: +``` +apt-get update +apt-get install ruby-dev +``` + +The Dpl provides support for vast number of services, including: Heroku, Cloud Foundry, AWS/S3, and more. +To use it simply define provider and any additional parameters required by the provider. + +For example if you want to use it to deploy your application to heroku, you need to specify `heroku` as provider, specify `api-key` and `app`. +There's more and all possible parameters can be found here: https://github.com/travis-ci/dpl#heroku + +``` +staging: + type: deploy + - gem install dpl + - dpl --provider=heroku --app=my-app-staging --api-key=$HEROKU_STAGING_API_KEY +``` + +In the above example we use Dpl to deploy `my-app-staging` to Heroku server with api-key stored in `HEROKU_STAGING_API_KEY` secure variable. + +To use different provider take a look at long list of [Supported Providers](https://github.com/travis-ci/dpl#supported-providers). + +### Using Dpl with Docker +When you use GitLab Runner you most likely configured it to use your server's shell commands. +This means that all commands are run in context of local user (ie. gitlab_runner or gitlab_ci_multi_runner). +It also means that most probably in your Docker container you don't have the Ruby runtime installed. +You will have to install it: +``` +staging: + type: deploy + - apt-get update -yq + - apt-get install -y ruby-dev + - gem install dpl + - dpl --provider=heroku --app=my-app-staging --api-key=$HEROKU_STAGING_API_KEY + only: + - master +``` + +The first line `apt-get update -yq` updates the list of available packages, where second `apt-get install -y ruby-dev` install `Ruby` runtime on system. +The above example is valid for all Debian-compatible systems. + +### Usage in staging and production +It's pretty common in developer workflow to have staging (development) and production environment. +If we consider above example: we would like to deploy `master` branch to `staging` and `all tags` to `production` environment. +The final `.gitlab-ci.yml` for that setup would look like this: + +``` +staging: + type: deploy + - gem install dpl + - dpl --provider=heroku --app=my-app-staging --api-key=$HEROKU_STAGING_API_KEY + only: + - master + +production: + type: deploy + - gem install dpl + - dpl --provider=heroku --app=my-app-production --api-key=$HEROKU_PRODUCTION_API_KEY + only: + - tags +``` + +We created two deploy jobs that are executed on different events: +1. `staging` is executed for all commits that were pushed to `master` branch, +2. `production` is executed for all pushed tags. + +We also use two secure variables: +1. `HEROKU_STAGING_API_KEY` - Heroku API key used to deploy staging app, +2. `HEROKU_PRODUCTION_API_KEY` - Heroku API key used to deploy production app. + +### Storing API keys +In GitLab CI 7.12 a new feature was introduced: Secure Variables. +Secure Variables can added by going to `Project > Variables > Add Variable`. +**This feature requires `gitlab-runner` with version equal or greater than 0.4.0.** +The variables that are defined in the project settings are send along with the build script to the runner. +The secure variables are stored out of the repository. Never store secrets in your projects' .gitlab-ci.yml. +It is also important that secret's value is hidden in the build log. + +You access added variable by prefixing it's name with `$` (on non-Windows runners) or `%` (for Windows Batch runners): +1. `$SECRET_VARIABLE` - use it for non-Windows runners +2. `%SECRET_VARIABLE%` - use it for Windows Batch runners diff --git a/doc/ci/docker/README.md b/doc/ci/docker/README.md new file mode 100644 index 00000000000..84eaf29efd1 --- /dev/null +++ b/doc/ci/docker/README.md @@ -0,0 +1,4 @@ +# Docker integration + ++ [Using Docker Images](using_docker_images.md) ++ [Using Docker Build](using_docker_build.md)
\ No newline at end of file diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md new file mode 100644 index 00000000000..5af27470d2f --- /dev/null +++ b/doc/ci/docker/using_docker_build.md @@ -0,0 +1,111 @@ +# Using Docker Build + +GitLab CI allows you to use Docker Engine to build and test docker-based projects. + +**This also allows to you to use `docker-compose` and other docker-enabled tools.** + +This is one of new trends in Continuous Integration/Deployment to: + +1. create application image, +1. run test against created image, +1. push image to remote registry, +1. deploy server from pushed image + +It's also useful in case when your application already has the `Dockerfile` that can be used to create and test image: +```bash +$ docker build -t my-image dockerfiles/ +$ docker run my-docker-image /script/to/run/tests +$ docker tag my-image my-registry:5000/my-image +$ docker push my-registry:5000/my-image +``` + +However, this requires special configuration of GitLab Runner to enable `docker` support during build. +**This requires running GitLab Runner in privileged mode which can be harmful when untrusted code is run.** + +There are two methods to enable the use of `docker build` and `docker run` during build. + +## 1. Use shell executor + +The simplest approach is to install GitLab Runner in `shell` execution mode. +GitLab Runner then executes build scripts as `gitlab-runner` user. + +1. Install [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/#installation). + +1. During GitLab Runner installation select `shell` as method of executing build scripts or use command: + + ```bash + $ sudo gitlab-runner register -n \ + --url http://gitlab.com/ci \ + --token RUNNER_TOKEN \ + --executor shell + --description "My Runner" + ``` + +2. Install Docker on server. + + For more information how to install Docker on different systems checkout the [Supported installations](https://docs.docker.com/installation/). + +3. Add `gitlab-runner` user to `docker` group: + + ```bash + $ sudo usermod -aG docker gitlab-runner + ``` + +4. Verify that `gitlab-runner` has access to Docker: + + ```bash + $ sudo -u gitlab-runner -H docker info + ``` + + You can now verify that everything works by adding `docker info` to `.gitlab-ci.yml`: + ```yaml + before_script: + - docker info + + build_image: + script: + - docker build -t my-docker-image . + - docker run my-docker-image /script/to/run/tests + ``` + +5. You can now use `docker` command and install `docker-compose` if needed. + +6. However, by adding `gitlab-runner` to `docker` group you are effectively granting `gitlab-runner` full root permissions. +For more information please checkout [On Docker security: `docker` group considered harmful](https://www.andreas-jung.com/contents/on-docker-security-docker-group-considered-harmful). + +## 2. Use docker-in-docker executor + +Second approach is to use special Docker image with all tools installed (`docker` and `docker-compose`) and run build script in context of that image in privileged mode. +In order to do that follow the steps: + +1. Install [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/#installation). + +1. Register GitLab Runner from command line to use `docker` and `privileged` mode: + + ```bash + $ sudo gitlab-runner register -n \ + --url http://gitlab.com/ci \ + --token RUNNER_TOKEN \ + --executor docker \ + --description "My Docker Runner" \ + --docker-image "gitlab/dind:latest" \ + --docker-privileged + ``` + + The above command will register new Runner to use special [gitlab/dind](https://registry.hub.docker.com/u/gitlab/dind/) image which is provided by GitLab Inc. + The image at the start runs Docker daemon in [docker-in-docker](https://blog.docker.com/2013/09/docker-can-now-run-within-docker/) mode. + +1. You can now use `docker` from build script: + + ```yaml + before_script: + - docker info + + build_image: + script: + - docker build -t my-docker-image . + - docker run my-docker-image /script/to/run/tests + ``` + +1. However, by enabling `--docker-privileged` you are effectively disables all security mechanisms of containers and exposing your host to privilege escalation which can lead to container breakout. +For more information, check out [Runtime privilege](https://docs.docker.com/reference/run/#runtime-privilege-linux-capabilities-and-lxc-configuration).
\ No newline at end of file diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md new file mode 100644 index 00000000000..191e3a8144d --- /dev/null +++ b/doc/ci/docker/using_docker_images.md @@ -0,0 +1,203 @@ +# Using Docker Images +GitLab CI can use [Docker Engine](https://www.docker.com/) to build projects. + +Docker is an open-source project that allows to use predefined images to run applications +in independent "containers" that are run within a single Linux instance. +[Docker Hub](https://registry.hub.docker.com/) have rich database of built images that can be used to build applications. + +Docker when used with GitLab CI runs each build in separate and isolated container using predefined image and always from scratch. +It makes it easier to have simple and reproducible build environment that can also be run on your workstation. +This allows you to test all commands from your shell, rather than having to test them on a CI server. + +### Register Docker runner +To use GitLab Runner with Docker you need to register new runner to use `docker` executor: + +```bash +gitlab-ci-multi-runner register \ + --url "https://gitlab.com/" \ + --registration-token "PROJECT_REGISTRATION_TOKEN" \ + --description "docker-ruby-2.1" \ + --executor "docker" \ + --docker-image ruby:2.1 \ + --docker-postgres latest \ + --docker-mysql latest +``` + +**The registered runner will use `ruby:2.1` image and will run two services (`postgres:latest` and `mysql:latest`) that will be accessible for time of the build.** + +### What is image? +The image is the name of any repository that is present in local Docker Engine or any repository that can be found at [Docker Hub](https://registry.hub.docker.com/). +For more information about the image and Docker Hub please read the [Docker Fundamentals](https://docs.docker.com/introduction/understanding-docker/). + +### What is service? +Service is just another image that is run for time of your build and is linked to your build. This allows you to access the service image during build time. +The service image can run any application, but most common use case is to run some database container, ie.: `mysql`. +It's easier and faster to use existing image, run it as additional container than install `mysql` every time project is built. + +#### How is service linked to the build? +There's good document that describes how Docker linking works: [Linking containers together](https://docs.docker.com/userguide/dockerlinks/). +To summarize: if you add `mysql` as service to your application, the image will be used to create container that is linked to build container. +The service container for MySQL will be accessible under hostname `mysql`. +So, **to access your database service you have to connect to host: `mysql` instead of socket or `localhost`**. + +### How to use other images as services? +You are not limited to have only database services. +You can hand modify `config.toml` to add any image as service found at [Docker Hub](https://registry.hub.docker.com/). +Look for `[runners.docker]` section: +``` +[runners.docker] + image = "ruby:2.1" + services = ["mysql:latest", "postgres:latest"] +``` + +For example you need `wordpress` instance to test some API integration with `Wordpress`. +You can for example use this image: [tutum/wordpress](https://registry.hub.docker.com/u/tutum/wordpress/). +This is image that have fully preconfigured `wordpress` and have `MySQL` server built-in: +``` +[runners.docker] + image = "ruby:2.1" + services = ["mysql:latest", "postgres:latest", "tutum/wordpress:latest"] +``` + +Next time when you run your application the `tutum/wordpress` will be started +and you will have access to it from your build container under hostname: `tutum_wordpress`. + +Alias hostname for the service is made from the image name: +1. Everything after `:` is stripped, +2. '/' is replaced to `_`. + +### Configuring services +Many services accept environment variables, which allow you to easily change database names or set account names depending on the environment. + +GitLab Runner 0.5.0 and up passes all YAML-defined variables to created service containers. + +1. To configure database name for [postgres](https://registry.hub.docker.com/u/library/postgres/) service, +you need to set POSTGRES_DB. + + ```yaml + services: + - postgres + + variables: + POSTGRES_DB: gitlab + ``` + +1. To use [mysql](https://registry.hub.docker.com/u/library/mysql/) service with empty password for time of build, +you need to set MYSQL_ALLOW_EMPTY_PASSWORD. + + ```yaml + services: + - mysql + + variables: + MYSQL_ALLOW_EMPTY_PASSWORD: yes + ``` + +For other possible configuration variables check the +https://registry.hub.docker.com/u/library/mysql/ or https://registry.hub.docker.com/u/library/postgres/ +or README page for any other Docker image. + +**Note: All variables will passed to all service containers. It's not designed to distinguish which variable should go where.** + +### Overwrite image and services +It's possible to overwrite `docker-image` and specify services from `.gitlab-ci.yml`. +If you add to your YAML the `image` and the `services` these parameters +be used instead of the ones that were specified during runner's registration. +``` +image: ruby:2.2 +services: + - postgres:9.3 +before_install: + - bundle install + +test: + script: + - bundle exec rake spec +``` + +It's possible to define image and service per-job: +``` +before_install: + - bundle install + +test:2.1: + image: ruby:2.1 + services: + - postgres:9.3 + script: + - bundle exec rake spec + +test:2.2: + image: ruby:2.2 + services: + - postgres:9.4 + script: + - bundle exec rake spec +``` + +#### How to enable overwriting? +To enable overwriting you have to **enable it first** (it's disabled by default for security reasons). +You can do that by hand modifying runner configuration: `config.toml`. +Please go to section where is `[runners.docker]` definition for your runner. +Add `allowed_images` and `allowed_services` to specify what images are allowed to be picked from `.gitlab-ci.yml`: +``` +[runners.docker] + image = "ruby:2.1" + allowed_images = ["ruby:*", "python:*"] + allowed_services = ["mysql:*", "redis:*"] +``` +This enables you to use in your `.gitlab-ci.yml` any image that matches above wildcards. +You will be able to pick only `ruby` and `python` images. +The same rule can be applied to limit services. + +If you are courageous enough, you can make it fully open and accept everything: +``` +[runners.docker] + image = "ruby:2.1" + allowed_images = ["*", "*/*"] + allowed_services = ["*", "*/*"] +``` + +**It the feature is not enabled, or image isn't allowed the error message will be put into the build log.** + +### How Docker integration works +1. Create any service container: `mysql`, `postgresql`, `mongodb`, `redis`. +1. Create cache container to store all volumes as defined in `config.toml` and `Dockerfile` of build image (`ruby:2.1` as in above example). +1. Create build container and link any service container to build container. +1. Start build container and send build script to the container. +1. Run build script. +1. Checkout code in: `/builds/group-name/project-name/`. +1. Run any step defined in `.gitlab-ci.yml`. +1. Check exit status of build script. +1. Remove build container and all created service containers. + +### How to debug a build locally +1. Create a file with build script: +```bash +$ cat <<EOF > build_script +git clone https://gitlab.com/gitlab-org/gitlab-ci-multi-runner.git /builds/gitlab-org/gitlab-ci-multi-runner +cd /builds/gitlab-org/gitlab-ci-multi-runner +make <- or any other build step +EOF +``` + +1. Create service containers: +``` +$ docker run -d -n service-mysql mysql:latest +$ docker run -d -n service-postgres postgres:latest +``` +This will create two service containers (MySQL and PostgreSQL). + +1. Create a build container and execute script in its context: +``` +$ cat build_script | docker run -n build -i -l mysql:service-mysql -l postgres:service-postgres ruby:2.1 /bin/bash +``` +This will create build container that has two service containers linked. +The build_script is piped using STDIN to bash interpreter which executes the build script in container. + +1. At the end remove all containers: +``` +docker rm -f -v build service-mysql service-postgres +``` +This will forcefully (the `-f` switch) remove build container and service containers +and all volumes (the `-v` switch) that were created with the container creation. diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md new file mode 100644 index 00000000000..e0b9fa0e25d --- /dev/null +++ b/doc/ci/examples/README.md @@ -0,0 +1,5 @@ +# Build script examples + ++ [Test and deploy Ruby Application to Heroku](test-and-deploy-ruby-application-to-heroku.md) ++ [Test and deploy Python Application to Heroku](test-and-deploy-python-application-to-heroku.md) ++ [Test Clojure applications](examples/test-clojure-application.md) diff --git a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md new file mode 100644 index 00000000000..036b03dd6b9 --- /dev/null +++ b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md @@ -0,0 +1,72 @@ +## Test and Deploy a python application +This example will guide you how to run tests in your Python application and deploy it automatically as Heroku application. + +You can checkout the example [source](https://gitlab.com/ayufan/python-getting-started) and check [CI status](https://ci.gitlab.com/projects/4080). + +### Configure project +This is what the `.gitlab-ci.yml` file looks like for this project: +```yaml +test: + script: + # this configures django application to use attached postgres database that is run on `postgres` host + - export DATABASE_URL=postgres://postgres:@postgres:5432/python-test-app + - apt-get update -qy + - apt-get install -y python-dev python-pip + - pip install -r requirements.txt + - python manage.py test + +staging: + type: deploy + script: + - apt-get update -qy + - apt-get install -y ruby-dev + - gem install dpl + - dpl --provider=heroku --app=gitlab-ci-python-test-staging --api-key=$HEROKU_STAGING_API_KEY + only: + - master + +production: + type: deploy + script: + - apt-get update -qy + - apt-get install -y ruby-dev + - gem install dpl + - dpl --provider=heroku --app=gitlab-ci-python-test-prod --api-key=$HEROKU_PRODUCTION_API_KEY + only: + - tags +``` + +This project has three jobs: +1. `test` - used to test rails application, +2. `staging` - used to automatically deploy staging environment every push to `master` branch +3. `production` - used to automatically deploy production environmnet for every created tag + +### Store API keys +You'll need to create two variables in `Project > Variables`: +1. `HEROKU_STAGING_API_KEY` - Heroku API key used to deploy staging app, +2. `HEROKU_PRODUCTION_API_KEY` - Heroku API key used to deploy production app. + +Find your Heroku API key in [Manage Account](https://dashboard.heroku.com/account). + +### Create Heroku application +For each of your environments, you'll need to create a new Heroku application. +You can do this through the [Dashboard](https://dashboard.heroku.com/). + +### Create runner +First install [Docker Engine](https://docs.docker.com/installation/). +To build this project you also need to have [GitLab Runner](https://about.gitlab.com/gitlab-ci/#gitlab-runner). +You can use public runners available on `gitlab.com/ci`, but you can register your own: +``` +gitlab-ci-multi-runner register \ + --non-interactive \ + --url "https://gitlab.com/ci/" \ + --registration-token "PROJECT_REGISTRATION_TOKEN" \ + --description "python-3.2" \ + --executor "docker" \ + --docker-image python:3.2 \ + --docker-postgres latest +``` + +With the command above, you create a runner that uses [python:3.2](https://registry.hub.docker.com/u/library/python/) image and uses [postgres](https://registry.hub.docker.com/u/library/postgres/) database. + +To access PostgreSQL database you need to connect to `host: postgres` as user `postgres` without password. diff --git a/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md new file mode 100644 index 00000000000..d2a872f1934 --- /dev/null +++ b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md @@ -0,0 +1,67 @@ +## Test and Deploy a ruby application +This example will guide you how to run tests in your Ruby application and deploy it automatiacally as Heroku application. + +You can checkout the example [source](https://gitlab.com/ayufan/ruby-getting-started) and check [CI status](https://ci.gitlab.com/projects/4050). + +### Configure project +This is what the `.gitlab-ci.yml` file looks like for this project: +```yaml +test: + script: + - apt-get update -qy + - apt-get install -y nodejs + - bundle install --path /cache + - bundle exec rake db:create RAILS_ENV=test + - bundle exec rake test + +staging: + type: deploy + script: + - gem install dpl + - dpl --provider=heroku --app=gitlab-ci-ruby-test-staging --api-key=$HEROKU_STAGING_API_KEY + only: + - master + +production: + type: deploy + script: + - gem install dpl + - dpl --provider=heroku --app=gitlab-ci-ruby-test-prod --api-key=$HEROKU_PRODUCTION_API_KEY + only: + - tags +``` + +This project has three jobs: +1. `test` - used to test rails application, +2. `staging` - used to automatically deploy staging environment every push to `master` branch +3. `production` - used to automatically deploy production environmnet for every created tag + +### Store API keys +You'll need to create two variables in `Project > Variables`: +1. `HEROKU_STAGING_API_KEY` - Heroku API key used to deploy staging app, +2. `HEROKU_PRODUCTION_API_KEY` - Heroku API key used to deploy production app. + +Find your Heroku API key in [Manage Account](https://dashboard.heroku.com/account). + +### Create Heroku application +For each of your environments, you'll need to create a new Heroku application. +You can do this through the [Dashboard](https://dashboard.heroku.com/). + +### Create runner +First install [Docker Engine](https://docs.docker.com/installation/). +To build this project you also need to have [GitLab Runner](https://about.gitlab.com/gitlab-ci/#gitlab-runner). +You can use public runners available on `gitlab.com/ci`, but you can register your own: +``` +gitlab-ci-multi-runner register \ + --non-interactive \ + --url "https://gitlab.com/ci/" \ + --registration-token "PROJECT_REGISTRATION_TOKEN" \ + --description "ruby-2.1" \ + --executor "docker" \ + --docker-image ruby:2.1 \ + --docker-postgres latest +``` + +With the command above, you create a runner that uses [ruby:2.1](https://registry.hub.docker.com/u/library/ruby/) image and uses [postgres](https://registry.hub.docker.com/u/library/postgres/) database. + +To access PostgreSQL database you need to connect to `host: postgres` as user `postgres` without password.
\ No newline at end of file diff --git a/doc/ci/examples/test-clojure-application.md b/doc/ci/examples/test-clojure-application.md new file mode 100644 index 00000000000..eaee94a10f1 --- /dev/null +++ b/doc/ci/examples/test-clojure-application.md @@ -0,0 +1,35 @@ +## Test Clojure applications + +This example will guide you how to run tests in your Clojure application. + +You can checkout the example [source](https://gitlab.com/dzaporozhets/clojure-web-application) and check [CI status](https://ci.gitlab.com/projects/6306). + +### Configure project + +This is what the `.gitlab-ci.yml` file looks like for this project: + +```yaml +variables: + POSTGRES_DB: sample-test + DATABASE_URL: "postgresql://postgres@postgres:5432/sample-test" + +before_script: + - apt-get update -y + - apt-get install default-jre postgresql-client -y + - wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein + - chmod a+x lein + - export LEIN_ROOT=1 + - PATH=$PATH:. + - lein deps + - lein migratus migrate + +test: + script: + - lein test +``` + +In before script we install JRE and [Leiningen](http://leiningen.org/). +Sample project uses [migratus](https://github.com/yogthos/migratus) library to manage database migrations. +So we added database migration as last step of `before_script` section + +You can use public runners available on `gitlab.com` for testing your application with such configuration. diff --git a/doc/ci/permissions/README.md b/doc/ci/permissions/README.md new file mode 100644 index 00000000000..d77061c14cd --- /dev/null +++ b/doc/ci/permissions/README.md @@ -0,0 +1,24 @@ +# Users Permissions + +GitLab CI relies on user's role on the GitLab. There are three permissions levels on GitLab CI: admin, master, developer, other. + +Admin user can perform any actions on GitLab CI in scope of instance and project. Also user with admin permission can use admin interface. + + + + +| Action | Guest, Reporter | Developer | Master | Admin | +|---------------------------------------|-----------------|-------------|----------|--------| +| See commits and builds | ✓ | ✓ | ✓ | ✓ | +| Retry or cancel build | | ✓ | ✓ | ✓ | +| Remove project | | | ✓ | ✓ | +| Create project | | | ✓ | ✓ | +| Change project configuration | | | ✓ | ✓ | +| Add specific runners | | | ✓ | ✓ | +| Add shared runners | | | | ✓ | +| See events in the system | | | | ✓ | +| Admin interface | | | | ✓ | + + + + diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md new file mode 100644 index 00000000000..a87a1f806fc --- /dev/null +++ b/doc/ci/quick_start/README.md @@ -0,0 +1,119 @@ +# Quick Start + +To start building projects with GitLab CI a few steps needs to be done. + +## 1. Install GitLab and CI + +First you need to have a working GitLab and GitLab CI instance. + +You can omit this step if you use [GitLab.com](http://GitLab.com/). + +## 2. Create repository on GitLab + +Once you login on your GitLab add a new repository where you will store your source code. +Push your application to that repository. + +## 3. Add project to CI + +The next part is to login to GitLab CI. +Point your browser to the URL you have set GitLab or use [gitlab.com/ci](http://gitlab.com/ci/). + +On the first screen you will see a list of GitLab's projects that you have access to: + + + +Click **Add Project to CI**. +This will create project in CI and authorize GitLab CI to fetch sources from GitLab. + +> GitLab CI creates unique token that is used to configure GitLab CI service in GitLab. +> This token allows to access GitLab's repository and configures GitLab to trigger GitLab CI webhook on **Push events** and **Tag push events**. +> You can see that token by going to Project's Settings > Services > GitLab CI. +> You will see there token, the same token is assigned in GitLab CI settings of project. + +## 4. Create project's configuration - .gitlab-ci.yml + +The next: You have to define how your project will be built. +GitLab CI uses [YAML](https://en.wikipedia.org/wiki/YAML) file to store build configuration. +You need to create `.gitlab-ci.yml` in root directory of your repository: + +```yaml +before_script: + - bundle install + +rspec: + script: + - bundle exec rspec + +rubocop: + script: + - bundle exec rubocop +``` + +This is the simplest possible build configuration that will work for most Ruby applications: +1. Define two jobs `rspec` and `rubocop` with two different commands to be executed. +1. Before every job execute commands defined by `before_script`. + +The `.gitlab-ci.yml` defines set of jobs with constrains how and when they should be run. +The jobs are defined as top-level elements with name and always have to contain the `script`. +Jobs are used to create builds, which are then picked by [runners](../runners/README.md) and executed within environment of the runner. +What is important that each job is run independently from each other. + +For more information and complete `.gitlab-ci.yml` syntax, please check the [Configuring project (.gitlab-ci.yml)](../yaml/README.md). + +## 5. Add file and push .gitlab-ci.yml to repository + +Once you created `.gitlab-ci.yml` you should add it to git repository and push it to GitLab. + +```bash +git add .gitlab-ci.yml +git commit +git push origin master +``` + +If you refresh the project's page on GitLab CI you will notice a one new commit: + + + +However the commit has status **pending** which means that commit was not yet picked by runner. + +## 6. Configure runner + +In GitLab CI, Runners run your builds. +A runner is a machine (can be virtual, bare-metal or VPS) that picks up builds through the coordinator API of GitLab CI. + +A runner can be specific to a certain project or serve any project in GitLab CI. +A runner that serves all projects is called a shared runner. +More information about different runner types can be found in [Configuring runner](../runners/README.md). + +To check if you have runners assigned to your project go to **Runners**. You will find there information how to setup project specific runner: + +1. Install GitLab Runner software. Checkout the [GitLab Runner](https://about.gitlab.com/gitlab-ci/#gitlab-runner) section to install it. +1. Specify following URL during runner setup: https://gitlab.com/ci/ +1. Use the following registration token during setup: TOKEN + +If you do it correctly your runner should be shown under **Runners activated for this project**: + + + +### Shared runners + +If you use [gitlab.com/ci](http://gitlab.com/ci/) you can use **Shared runners** provided by GitLab Inc. +These are special virtual machines that are run on GitLab's infrastructure that can build any project. +To enable **Shared runners** you have to go to **Runners** and click **Enable shared runners** for this project. + +## 7. Check status of commit + +If everything went OK and you go to commit, the status of the commit should change from **pending** to either **running**, **success** or **failed**. + + + +You can click **Build ID** to view build log for specific job. + +## 8. Congratulations! + +You managed to build your first project using GitLab CI. +You may need to tune your `.gitlab-ci.yml` file to implement build plan for your project. +A few examples how it can be done you can find on [Examples](../examples/README.md) page. + +GitLab CI also offers **the Lint** tool to verify validity of your `.gitlab-ci.yml` which can be useful to troubleshoot potential problems. +The Lint is available from project's settings or by adding `/lint` to GitLab CI url. diff --git a/doc/ci/quick_start/build_status.png b/doc/ci/quick_start/build_status.png Binary files differnew file mode 100644 index 00000000000..333259e6acd --- /dev/null +++ b/doc/ci/quick_start/build_status.png diff --git a/doc/ci/quick_start/commit_status.png b/doc/ci/quick_start/commit_status.png Binary files differnew file mode 100644 index 00000000000..725b79e6f91 --- /dev/null +++ b/doc/ci/quick_start/commit_status.png diff --git a/doc/ci/quick_start/new_commit.png b/doc/ci/quick_start/new_commit.png Binary files differnew file mode 100644 index 00000000000..3839e893c17 --- /dev/null +++ b/doc/ci/quick_start/new_commit.png diff --git a/doc/ci/quick_start/projects.png b/doc/ci/quick_start/projects.png Binary files differnew file mode 100644 index 00000000000..0b3430a69db --- /dev/null +++ b/doc/ci/quick_start/projects.png diff --git a/doc/ci/quick_start/runners.png b/doc/ci/quick_start/runners.png Binary files differnew file mode 100644 index 00000000000..25b4046bc00 --- /dev/null +++ b/doc/ci/quick_start/runners.png diff --git a/doc/ci/quick_start/runners_activated.png b/doc/ci/quick_start/runners_activated.png Binary files differnew file mode 100644 index 00000000000..c934bd12f41 --- /dev/null +++ b/doc/ci/quick_start/runners_activated.png diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md new file mode 100644 index 00000000000..68dcfe23ffb --- /dev/null +++ b/doc/ci/runners/README.md @@ -0,0 +1,145 @@ +# Runners + +In GitLab CI, Runners run your [yaml](../yaml/README.md). +A runner is an isolated (virtual) machine that picks up builds +through the coordinator API of GitLab CI. + +A runner can be specific to a certain project or serve any project +in GitLab CI. A runner that serves all projects is called a shared runner. + +## Shared vs. Specific Runners + +A runner that is specific only runs for the specified project. A shared runner +can run jobs for every project that has enabled the option +`Allow shared runners`. + +**Shared runners** are useful for jobs that have similar requirements, +between multiple projects. Rather than having multiple runners idling for +many projects, you can have a single or a small number of runners that handle +multiple projects. This makes it easier to maintain and update runners. + +**Specific runners** are useful for jobs that have special requirements or for +projects with a very demand. If a job has certain requirements, you can set +up the specific runner with this in mind, while not having to do this for all +runners. For example, if you want to deploy a certain project, you can setup +a specific runner to have the right credentials for this. + +Projects with high demand of CI activity can also benefit from using specific runners. +By having dedicated runners you are guaranteed that the runner is not being held +up by another project's jobs. + +You can set up a specific runner to be used by multiple projects. The difference +with a shared runner is that you have to enable each project explicitly for +the runner to be able to run its jobs. + +Specific runners do not get shared with forked projects automatically. +A fork does copy the CI settings (jobs, allow shared, etc) of the cloned repository. + +# Creating and Registering a Runner + +There are several ways to create a runner. Only after creation, upon +registration its status as Shared or Specific is determined. + +[See the documentation for](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/#installation) +the different methods of installing a Runner instance. + +After installing the runner, you can either register it as `Shared` or as `Specific`. +You can only register a Shared Runner if you have admin access to the GitLab instance. + +## Registering a Shared Runner + +You can only register a shared runner if you are an admin on the linked +GitLab instance. + +Grab the shared-runner token on the `admin/runners` page of your GitLab CI +instance. + + + +Now simply register the runner as any runner: + +``` +sudo gitlab-runner register +``` + +Note that you will have to enable `Allows shared runners` for each project +that you want to make use of a shared runner. This is by default `off`. + +## Registering a Specific Runner + +Registering a specific can be done in two ways: + +1. Creating a runner with the project registration token +1. Converting a shared runner into a specific runner (one-way, admin only) + +There are several ways to create a runner instance. The steps below only +concern registering the runner on GitLab CI. + +### Registering a Specific Runner with a Project Registration token + +To create a specific runner without having admin rights to the GitLab instance, +visit the project you want to make the runner work for in GitLab CI. + +Click on the runner tab and use the registration token you find there to +setup a specific runner for this project. + + + +To register the runner, run the command below and follow instructions: + +``` +sudo gitlab-runner register +``` + +### Making an existing Shared Runner Specific + +If you are an admin on your GitLab instance, +you can make any shared runner a specific runner, _but you can not +make a specific runner a shared runner_. + +To make a shared runner specific, go to the runner page (`/admin/runners`) +and find your runner. Add any projects on the left to make this runner +run exclusively for these projects, therefore making it a specific runner. + + + +## Using Shared Runners Effectively + +If you are planning to use shared runners, there are several things you +should keep in mind. + +### Use Tags + +You must setup a runner to be able to run all the different types of jobs +that it may encounter on the projects it's shared over. This would be +problematic for large amounts of projects, if it wasn't for tags. + +By tagging a Runner for the types of jobs it can handle, you can make sure +shared runners will only run the jobs they are equipped to run. + +For instance, at GitLab we have runners tagged with "rails" if they contain +the appropriate dependencies to run Rails test suites. + +### Be Careful with Sensitive Information + +If you can run a build on a runner, you can get access to any code it runs +and get the token of the runner. With shared runners, this means that anyone +that runs jobs on the runner, can access anyone else's code that runs on the runner. + +In addition, because you can get access to the runner token, it is possible +to create a clone of a runner and submit false builds, for example. + +The above is easily avoided by restricting the usage of shared runners +on large public GitLab instances and controlling access to your GitLab instance. + +### Forks + +Whenever a project is forked, it copies the settings of the jobs that relate +to it. This means that if you have shared runners setup for a project and +someone forks that project, the shared runners will also serve jobs of this +project. + +# Attack vectors in runners + +Mentioned briefly earlier, but the following things of runners can be exploited. +We're always looking for contributions that can mitigate these [Security Considerations](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/security/index.md). diff --git a/doc/ci/runners/project_specific.png b/doc/ci/runners/project_specific.png Binary files differnew file mode 100644 index 00000000000..f51ea694e78 --- /dev/null +++ b/doc/ci/runners/project_specific.png diff --git a/doc/ci/runners/shared_runner.png b/doc/ci/runners/shared_runner.png Binary files differnew file mode 100644 index 00000000000..9755144eb08 --- /dev/null +++ b/doc/ci/runners/shared_runner.png diff --git a/doc/ci/runners/shared_to_specific_admin.png b/doc/ci/runners/shared_to_specific_admin.png Binary files differnew file mode 100644 index 00000000000..44a4bef22f7 --- /dev/null +++ b/doc/ci/runners/shared_to_specific_admin.png diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md new file mode 100644 index 00000000000..04c6bf1e3a3 --- /dev/null +++ b/doc/ci/variables/README.md @@ -0,0 +1,95 @@ +## Variables +When receiving a build from GitLab CI, the runner prepares the build environment. +It starts by setting a list of **predefined variables** (Environment Variables) and a list of **user-defined variables** + +The variables can be overwritten. They take precedence over each other in this order: +1. Secure variables +1. YAML-defined variables +1. Predefined variables + +For example, if you define: +1. API_TOKEN=SECURE as Secure Variable +1. API_TOKEN=YAML as YAML-defined variable + +The API_TOKEN will take the Secure Variable value: `SECURE`. + +### Predefined variables (Environment Variables) + +| Variable | Description | +|-------------------------|-------------| +| **CI** | Mark that build is executed in CI environment | +| **GITLAB_CI** | Mark that build is executed in GitLab CI environment | +| **CI_SERVER** | Mark that build is executed in CI environment | +| **CI_SERVER_NAME** | CI server that is used to coordinate builds | +| **CI_SERVER_VERSION** | Not yet defined | +| **CI_SERVER_REVISION** | Not yet defined | +| **CI_BUILD_REF** | The commit revision for which project is built | +| **CI_BUILD_BEFORE_SHA** | The first commit that were included in push request | +| **CI_BUILD_REF_NAME** | The branch or tag name for which project is built | +| **CI_BUILD_ID** | The unique id of the current build that GitLab CI uses internally | +| **CI_BUILD_REPO** | The URL to clone the Git repository | +| **CI_PROJECT_ID** | The unique id of the current project that GitLab CI uses internally | +| **CI_PROJECT_DIR** | The full path where the repository is cloned and where the build is ran | + +Example values: + +```bash +export CI_BUILD_BEFORE_SHA="9df57456fa9de2a6d335ca5edf9750ed812b9df0" +export CI_BUILD_ID="50" +export CI_BUILD_REF="1ecfd275763eff1d6b4844ea3168962458c9f27a" +export CI_BUILD_REF_NAME="master" +export CI_BUILD_REPO="https://gitlab.com/gitlab-org/gitlab-ce.git" +export CI_PROJECT_DIR="/builds/gitlab-org/gitlab-ce" +export CI_PROJECT_ID="34" +export CI_SERVER="yes" +export CI_SERVER_NAME="GitLab CI" +export CI_SERVER_REVISION="" +export CI_SERVER_VERSION="" +``` + +### YAML-defined variables +**This feature requires GitLab Runner 0.5.0 or higher** + +GitLab CI allows you to add to `.gitlab-ci.yml` variables that are set in build environment. +The variables are stored in repository and are meant to store non-sensitive project configuration, ie. RAILS_ENV or DATABASE_URL. + +```yaml +variables: + DATABASE_URL: "postgres://postgres@postgres/my_database" +``` + +These variables can be later used in all executed commands and scripts. + +The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them. + +More information about Docker integration can be found in [Using Docker Images](../docker/using_docker_images.md). + +### User-defined variables (Secure Variables) +**This feature requires GitLab Runner 0.4.0 or higher** + +GitLab CI allows you to define per-project **Secure Variables** that are set in build environment. +The secure variables are stored out of the repository (the `.gitlab-ci.yml`). +These variables are securely stored in GitLab CI database and are hidden in the build log. +It's desired method to use them for storing passwords, secret keys or whatever you want. + +Secure Variables can added by going to `Project > Variables > Add Variable`. + +They will be available for all subsequent builds. + +### Use variables +The variables are set as environment variables in build environment and are accessible with normal methods that are used to access such variables. +In most cases the **bash** is used to execute build script. +To access variables (predefined and user-defined) in bash environment, prefix the variable name with `$`: +``` +job_name: + script: + - echo $CI_BUILD_ID +``` + +You can also list all environment variables with `export` command, +but be aware that this will also expose value of all **Secure Variables** in build log: +``` +job_name: + script: + - export +``` diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md new file mode 100644 index 00000000000..4caeccacb7f --- /dev/null +++ b/doc/ci/yaml/README.md @@ -0,0 +1,204 @@ +# Configuration of your builds with .gitlab-ci.yml +From version 7.12, GitLab CI uses a [YAML](https://en.wikipedia.org/wiki/YAML) file (**.gitlab-ci.yml**) for the project configuration. +It is placed in the root of your repository and contains definitions of how your project should be built. + +The YAML file defines a set of jobs with constraints stating when they should be run. +The jobs are defined as top-level elements with a name and always have to contain the `script` clause: + +```yaml +job1: + script: "execute-script-for-job1" + +job2: + script: "execute-script-for-job2" +``` + +The above example is the simplest possible CI configuration with two separate jobs, +where each of the jobs executes a different command. +Of course a command can execute code directly (`./configure;make;make install`) or run a script (`test.sh`) in the repository. + +Jobs are used to create builds, which are then picked up by [runners](../runners/README.md) and executed within the environment of the runner. +What is important, is that each job is run independently from each other. + +## .gitlab-ci.yml +The YAML syntax allows for using more complex job specifications than in the above example: + +```yaml +image: ruby:2.1 +services: + - postgres + +before_script: + - bundle_install + +stages: + - build + - test + - deploy + +job1: + stage: build + script: + - execute-script-for-job1 + only: + - master + tags: + - docker +``` + +There are a few `keywords` that can't be used as job names: + +| keyword | required | description | +|---------------|----------|-------------| +| image | optional | Use docker image, covered in [Use Docker](../docker/README.md) | +| services | optional | Use docker services, covered in [Use Docker](../docker/README.md) | +| stages | optional | Define build stages | +| types | optional | Alias for `stages` | +| before_script | optional | Define commands prepended for each job's script | +| variables | optional | Define build variables | + +### image and services +This allows to specify a custom Docker image and a list of services that can be used for time of the build. +The configuration of this feature is covered in separate document: [Use Docker](../docker/README.md). + +### before_script +`before_script` is used to define the command that should be run before all builds, including deploy builds. This can be an array or a multiline string. + +### stages +`stages` is used to define build stages that can be used by jobs. +The specification of `stages` allows for having flexible multi stage pipelines. + +The ordering of elements in `stages` defines the ordering of builds' execution: + +1. Builds of the same stage are run in parallel. +1. Builds of next stage are run after success. + +Let's consider the following example, which defines 3 stages: +``` +stages: + - build + - test + - deploy +``` + +1. First all jobs of `build` are executed in parallel. +1. If all jobs of `build` succeeds, the `test` jobs are executed in parallel. +1. If all jobs of `test` succeeds, the `deploy` jobs are executed in parallel. +1. If all jobs of `deploy` succeeds, the commit is marked as `success`. +1. If any of the previous jobs fails, the commit is marked as `failed` and no jobs of further stage are executed. + +There are also two edge cases worth mentioning: + +1. If no `stages` is defined in `.gitlab-ci.yml`, then by default the `build`, `test` and `deploy` are allowed to be used as job's stage by default. +2. If a job doesn't specify `stage`, the job is assigned the `test` stage. + +### types +Alias for [stages](#stages). + +### variables +**This feature requires `gitlab-runner` with version equal or greater than 0.5.0.** + +GitLab CI allows you to add to `.gitlab-ci.yml` variables that are set in build environment. +The variables are stored in repository and are meant to store non-sensitive project configuration, ie. RAILS_ENV or DATABASE_URL. + +```yaml +variables: + DATABASE_URL: "postgres://postgres@postgres/my_database" +``` + +These variables can be later used in all executed commands and scripts. + +The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them. + +## Jobs +`.gitlab-ci.yml` allows you to specify an unlimited number of jobs. +Each job has to have a unique `job_name`, which is not one of the keywords mentioned above. +A job is defined by a list of parameters that define the build behaviour. + +```yaml +job_name: + script: + - rake spec + - coverage + stage: test + only: + - master + except: + - develop + tags: + - ruby + - postgres + allow_failure: true +``` + +| keyword | required | description | +|---------------|----------|-------------| +| script | required | Defines a shell script which is executed by runner | +| stage | optional (default: test) | Defines a build stage | +| type | optional | Alias for `stage` | +| only | optional | Defines a list of git refs for which build is created | +| except | optional | Defines a list of git refs for which build is not created | +| tags | optional | Defines a list of tags which are used to select runner | +| allow_failure | optional | Allow build to fail. Failed build doesn't contribute to commit status | + +### script +`script` is a shell script which is executed by runner. The shell script is prepended with `before_script`. + +```yaml +job: + script: "bundle exec rspec" +``` + +This parameter can also contain several commands using an array: +```yaml +job: + script: + - uname -a + - bundle exec rspec +``` + +### stage +`stage` allows to group build into different stages. Builds of the same `stage` are executed in `parallel`. +For more info about the use of `stage` please check the [stages](#stages). + +### only and except +This are two parameters that allow for setting a refs policy to limit when jobs are built: +1. `only` defines the names of branches and tags for which job will be built. +2. `except` defines the names of branches and tags for which the job wil **not** be built. + +There are a few rules that apply to usage of refs policy: + +1. `only` and `except` are exclusive. If both `only` and `except` are defined in job specification only `only` is taken into account. +1. `only` and `except` allow for using the regexp expressions. +1. `only` and `except` allow for using special keywords: `branches` and `tags`. +These names can be used for example to exclude all tags and all branches. + +```yaml +job: + only: + - /^issue-.*$/ # use regexp + except: + - branches # use special keyword +``` + +### tags +`tags` is used to select specific runners from the list of all runners that are allowed to run this project. + +During registration of a runner, you can specify the runner's tags, ie.: `ruby`, `postgres`, `development`. +`tags` allow you to run builds with runners that have the specified tags assigned: + +``` +job: + tags: + - ruby + - postgres +``` + +The above specification will make sure that `job` is built by a runner that have `ruby` AND `postgres` tags defined. + +## Validate the .gitlab-ci.yml +Each instance of GitLab CI has an embedded debug tool called Lint. +You can find the link to the Lint in the project's settings page or use short url `/lint`. + +## Skipping builds +There is one more way to skip all builds, if your commit message contains tag [ci skip]. In this case, commit will be created but builds will be skipped
\ No newline at end of file diff --git a/doc/gitlab-basics/create-your-ssh-keys.md b/doc/gitlab-basics/create-your-ssh-keys.md index c8a73feb028..f31c353f2cf 100644 --- a/doc/gitlab-basics/create-your-ssh-keys.md +++ b/doc/gitlab-basics/create-your-ssh-keys.md @@ -10,11 +10,7 @@ After you confirm, go to GitLab and sign in to your account. ## Add your SSH Key -At the top right corner, click on "profile settings": - - - -On the left side menu click on "SSH Keys": +On the left side menu, click on "profile settings" and then click on "SSH Keys":  diff --git a/doc/hooks/custom_hooks.md b/doc/hooks/custom_hooks.md index f7d4f3de68b..548c484bc08 100644 --- a/doc/hooks/custom_hooks.md +++ b/doc/hooks/custom_hooks.md @@ -2,7 +2,7 @@ **Note: Custom git hooks must be configured on the filesystem of the GitLab server. Only GitLab server administrators will be able to complete these tasks. -Please explore webhooks as an option if you do not have filesystem access.** +Please explore webhooks as an option if you do not have filesystem access. For a user configurable Git Hooks interface, please see [GitLab Enterprise Edition Git Hooks](http://doc.gitlab.com/ee/git_hooks/git_hooks.html).** Git natively supports hooks that are executed on different actions. Examples of server-side git hooks include pre-receive, post-receive, and update. diff --git a/doc/incoming_email/README.md b/doc/incoming_email/README.md new file mode 100644 index 00000000000..01ab22321ed --- /dev/null +++ b/doc/incoming_email/README.md @@ -0,0 +1,207 @@ +# Reply by email + +GitLab can be set up to allow users to comment on issues and merge requests by replying to notification emails. + +## Get a mailbox + +Reply by email requires an IMAP-enabled email account, with a provider or server that supports [email sub-addressing](https://en.wikipedia.org/wiki/Email_address#Sub-addressing). Sub-addressing is a feature where any email to `user+some_arbitrary_tag@example.com` will end up in the mailbox for `user@example.com`, and is supported by providers such as Gmail, Yahoo! Mail, Outlook.com and iCloud, as well as the Postfix mail server which you can run on-premises. + +If you want to use Gmail with Reply by email, make sure you have [IMAP access enabled](https://support.google.com/mail/troubleshooter/1668960?hl=en#ts=1665018) and [allow less secure apps to access the account](https://support.google.com/accounts/answer/6010255). + +To set up a basic Postfix mail server with IMAP access on Ubuntu, follow [these instructions](./postfix.md). + +## Set it up + +In this example, we'll use the Gmail address `gitlab-incoming@gmail.com`. + +### Omnibus package installations + +1. Find the `incoming_email` section in `/etc/gitlab/gitlab.rb`, enable the feature, enter the email address including a placeholder for the `key` that references the item being replied to and fill in the details for your specific IMAP server and email account: + + ```ruby + gitlab_rails['incoming_email_enabled'] = true + gitlab_rails['incoming_email_address'] = "gitlab-incoming+%{key}@gmail.com" + gitlab_rails['incoming_email_host'] = "imap.gmail.com" # IMAP server host + gitlab_rails['incoming_email_port'] = 993 # IMAP server port + gitlab_rails['incoming_email_ssl'] = true # Whether the IMAP server uses SSL + gitlab_rails['incoming_email_email'] = "gitlab-incoming@gmail.com" # Email account username. Usually the full email address. + gitlab_rails['incoming_email_password'] = "password" # Email account password + gitlab_rails['incoming_email_mailbox_name'] = "inbox" # The name of the mailbox where incoming mail will end up. Usually "inbox". + ``` + + As mentioned, the part after `+` in the address is ignored, and any email sent here will end up in the mailbox for `gitlab-incoming@gmail.com`. + +1. Reconfigure GitLab for the changes to take effect: + + ```sh + sudo gitlab-ctl reconfigure + ``` + +1. Verify that everything is configured correctly: + + ```sh + sudo gitlab-rake gitlab:incoming_email:check + ``` + +1. Reply by email should now be working. + +### Installations from source + +1. Go to the GitLab installation directory: + + ```sh + cd /home/git/gitlab + ``` + +1. Find the `incoming_email` section in `config/gitlab.yml`, enable the feature and enter the email address including a placeholder for the `key` that references the item being replied to: + + ```sh + sudo editor config/gitlab.yml + ``` + + ```yaml + incoming_email: + enabled: true + address: "gitlab-incoming+%{key}@gmail.com" + ``` + + As mentioned, the part after `+` in the address is ignored, and any email sent here will end up in the mailbox for `gitlab-incoming@gmail.com`. + +2. Copy `config/mail_room.yml.example` to `config/mail_room.yml`: + + ```sh + sudo cp config/mail_room.yml.example config/mail_room.yml + ``` + +3. Uncomment the configuration options in `config/mail_room.yml` and fill in the details for your specific IMAP server and email account: + + ```sh + sudo editor config/mail_room.yml + ``` + + ```yaml + :mailboxes: + - + # IMAP server host + :host: "imap.gmail.com" + # IMAP server port + :port: 993 + # Whether the IMAP server uses SSL + :ssl: true + # Whether the IMAP server uses StartTLS + :start_tls: false + # Email account username. Usually the full email address. + :email: "gitlab-incoming@gmail.com" + # Email account password + :password: "[REDACTED]" + # The name of the mailbox where incoming mail will end up. Usually "inbox". + :name: "inbox" + # Always "sidekiq". + :delivery_method: sidekiq + # Always true. + :delete_after_delivery: true + :delivery_options: + # The URL to the Redis server used by Sidekiq. Should match the URL in config/resque.yml. + :redis_url: redis://localhost:6379 + # Always "resque:gitlab". + :namespace: resque:gitlab + # Always "incoming_email". + :queue: incoming_email + # Always "EmailReceiverWorker" + :worker: EmailReceiverWorker + ``` + +5. Edit the init script configuration at `/etc/default/gitlab` to enable `mail_room`: + + ```sh + sudo mkdir -p /etc/default + echo 'mail_room_enabled=true' | sudo tee -a /etc/default/gitlab + ``` + +6. Restart GitLab: + + ```sh + sudo service gitlab restart + ``` + +7. Verify that everything is configured correctly: + + ```sh + sudo -u git -H bundle exec rake gitlab:incoming_email:check RAILS_ENV=production + ``` + +8. Reply by email should now be working. + +### Development + +1. Go to the GitLab installation directory. + +1. Find the `incoming_email` section in `config/gitlab.yml`, enable the feature and enter the email address including a placeholder for the `key` that references the item being replied to: + + ```yaml + incoming_email: + enabled: true + address: "gitlab-incoming+%{key}@gmail.com" + ``` + + As mentioned, the part after `+` is ignored, and this will end up in the mailbox for `gitlab-incoming@gmail.com`. + +2. Copy `config/mail_room.yml.example` to `config/mail_room.yml`: + + ```sh + sudo cp config/mail_room.yml.example config/mail_room.yml + ``` + +3. Uncomment the configuration options in `config/mail_room.yml` and fill in the details for your specific IMAP server and email account: + + ```yaml + :mailboxes: + - + # IMAP server host + :host: "imap.gmail.com" + # IMAP server port + :port: 993 + # Whether the IMAP server uses SSL + :ssl: true + # Whether the IMAP server uses StartTLS + :start_tls: false + # Email account username. Usually the full email address. + :email: "gitlab-incoming@gmail.com" + # Email account password + :password: "[REDACTED]" + # The name of the mailbox where incoming mail will end up. Usually "inbox". + :name: "inbox" + # Always "sidekiq". + :delivery_method: sidekiq + # Always true. + :delete_after_delivery: true + :delivery_options: + # The URL to the Redis server used by Sidekiq. Should match the URL in config/resque.yml. + :redis_url: redis://localhost:6379 + # Always "resque:gitlab". + :namespace: resque:gitlab + # Always "incoming_email". + :queue: incoming_email + # Always "EmailReceiverWorker" + :worker: EmailReceiverWorker + ``` + +4. Uncomment the `mail_room` line in your `Procfile`: + + ```yaml + mail_room: bundle exec mail_room -q -c config/mail_room.yml + ``` + +6. Restart GitLab: + + ```sh + bundle exec foreman start + ``` + +7. Verify that everything is configured correctly: + + ```sh + bundle exec rake gitlab:incoming_email:check RAILS_ENV=development + ``` + +8. Reply by email should now be working. diff --git a/doc/incoming_email/postfix.md b/doc/incoming_email/postfix.md new file mode 100644 index 00000000000..18bf3db1744 --- /dev/null +++ b/doc/incoming_email/postfix.md @@ -0,0 +1,310 @@ +# Set up Postfix for Reply by email + +This document will take you through the steps of setting up a basic Postfix mail server with IMAP authentication on Ubuntu, to be used with Reply by email. + +The instructions make the assumption that you will be using the email address `incoming@gitlab.example.com`, that is, username `incoming` on host `gitlab.example.com`. Don't forget to change it to your actual host when executing the example code snippets. + +## Configure your server firewall + +1. Open up port 25 on your server so that people can send email into the server over SMTP. +2. If the mail server is different from the server running GitLab, open up port 143 on your server so that GitLab can read email from the server over IMAP. + +## Install packages + +1. Install the `postfix` package if it is not installed already: + + ```sh + sudo apt-get install postfix + ``` + + When asked about the environment, select 'Internet Site'. When asked to confirm the hostname, make sure it matches `gitlab.example.com`. + +1. Install the `mailutils` package. + + ```sh + sudo apt-get install mailutils + ``` + +## Create user + +1. Create a user for incoming email. + + ```sh + sudo useradd -m -s /bin/bash incoming + ``` + +1. Set a password for this user. + + ```sh + sudo passwd incoming + ``` + + Be sure not to forget this, you'll need it later. + +## Test the out-of-the-box setup + +1. Connect to the local SMTP server: + + ```sh + telnet localhost 25 + ``` + + You should see a prompt like this: + + ```sh + Trying 127.0.0.1... + Connected to localhost. + Escape character is '^]'. + 220 gitlab.example.com ESMTP Postfix (Ubuntu) + ``` + + If you get a `Connection refused` error instead, verify that `postfix` is running: + + ```sh + sudo postfix status + ``` + + If it is not, start it: + + ```sh + sudo postfix start + ``` + +1. Send the new `incoming` user a dummy email to test SMTP, by entering the following into the SMTP prompt: + + ``` + ehlo localhost + mail from: root@localhost + rcpt to: incoming@localhost + data + Subject: Re: Some issue + + Sounds good! + . + quit + ``` + + (Note: The `.` is a literal period on its own line) + +1. Check if the `incoming` user received the email: + + ```sh + su - incoming + mail + ``` + + You should see output like this: + + ``` + "/var/mail/incoming": 1 message 1 unread + >U 1 root@localhost 59/2842 Re: Some issue + ``` + + Quit the mail app: + + ```sh + q + ``` + +1. Log out of the `incoming` account and go back to being `root`: + + ```sh + logout + ``` + +## Configure Postfix to use Maildir-style mailboxes + +Courier, which we will install later to add IMAP authentication, requires mailboxes to have the Maildir format, rather than mbox. + +1. Configure Postfix to use Maildir-style mailboxes: + + ```sh + sudo postconf -e "home_mailbox = Maildir/" + ``` + +1. Restart Postfix: + + ```sh + sudo /etc/init.d/postfix restart + ``` + +1. Test the new setup: + + 1. Follow steps 1 and 2 of _[Test the out-of-the-box setup](#test-the-out-of-the-box-setup)_. + 2. Check if the `incoming` user received the email: + + ```sh + su - incoming + MAIL=/home/incoming/Maildir + mail + ``` + + You should see output like this: + + ``` + "/home/incoming/Maildir": 1 message 1 unread + >U 1 root@localhost 59/2842 Re: Some issue + ``` + + Quit the mail app: + + ```sh + q + ``` + +1. Log out of the `incoming` account and go back to being `root`: + + ```sh + logout + ``` + +## Install the Courier IMAP server + +1. Install the `courier-imap` package: + + ```sh + sudo apt-get install courier-imap + ``` + +## Configure Postfix to receive email from the internet + +1. Let Postfix know about the domains that it should consider local: + + ```sh + sudo postconf -e "mydestination = gitlab.example.com, localhost.localdomain, localhost" + ``` + +1. Let Postfix know about the IPs that it should consider part of the LAN: + + We'll assume `192.168.1.0/24` is your local LAN. You can safely skip this step if you don't have other machines in the same local network. + + ```sh + sudo postconf -e "mynetworks = 127.0.0.0/8, 192.168.1.0/24" + ``` + +1. Configure Postfix to receive mail on all interfaces, which includes the internet: + + ```sh + sudo postconf -e "inet_interfaces = all" + ``` + +1. Configure Postfix to use the `+` delimiter for sub-addressing: + + ```sh + sudo postconf -e "recipient_delimiter = +" + ``` + +1. Restart Postfix: + + ```sh + sudo service postfix restart + ``` + +## Test the final setup + +1. Test SMTP under the new setup: + + 1. Connect to the SMTP server: + + ```sh + telnet gitlab.example.com 25 + ``` + + You should see a prompt like this: + + ```sh + Trying 123.123.123.123... + Connected to gitlab.example.com. + Escape character is '^]'. + 220 gitlab.example.com ESMTP Postfix (Ubuntu) + ``` + + If you get a `Connection refused` error instead, make sure your firewall is setup to allow inbound traffic on port 25. + + 1. Send the `incoming` user a dummy email to test SMTP, by entering the following into the SMTP prompt: + + ``` + ehlo gitlab.example.com + mail from: root@gitlab.example.com + rcpt to: incoming@gitlab.example.com + data + Subject: Re: Some issue + + Sounds good! + . + quit + ``` + + (Note: The `.` is a literal period on its own line) + + 1. Check if the `incoming` user received the email: + + ```sh + su - incoming + MAIL=/home/incoming/Maildir + mail + ``` + + You should see output like this: + + ``` + "/home/incoming/Maildir": 1 message 1 unread + >U 1 root@gitlab.example.com 59/2842 Re: Some issue + ``` + + Quit the mail app: + + ```sh + q + ``` + + 1. Log out of the `incoming` account and go back to being `root`: + + ```sh + logout + ``` + +1. Test IMAP under the new setup: + + 1. Connect to the IMAP server: + + ```sh + telnet gitlab.example.com 143 + ``` + + You should see a prompt like this: + + ```sh + Trying 123.123.123.123... + Connected to mail.example.gitlab.com. + Escape character is '^]'. + - OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION] Courier-IMAP ready. Copyright 1998-2011 Double Precision, Inc. See COPYING for distribution information. + ``` + + 1. Sign in as the `incoming` user to test IMAP, by entering the following into the IMAP prompt: + + ``` + a login incoming PASSWORD + ``` + + Replace PASSWORD with the password you set on the `incoming` user earlier. + + You should see output like this: + + ``` + a OK LOGIN Ok. + ``` + + 1. Disconnect from the IMAP server: + + ```sh + a logout + ``` + +## Done! + +If all the tests were successful, Postfix is all set up and ready to receive email! Continue with the [Reply by email](./README.md) guide to configure GitLab. + +--------- + +_This document was adapted from https://help.ubuntu.com/community/PostfixBasicSetupHowto, by contributors to the Ubuntu documentation wiki._ diff --git a/doc/install/database_mysql.md b/doc/install/database_mysql.md index 362c492d0ac..c565e90da2f 100644 --- a/doc/install/database_mysql.md +++ b/doc/install/database_mysql.md @@ -36,7 +36,7 @@ We do not recommend using MySQL due to various issues. For example, case [(in)se mysql> CREATE DATABASE IF NOT EXISTS `gitlabhq_production` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`; # Grant the GitLab user necessary permissions on the database - mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, LOCK TABLES ON `gitlabhq_production`.* TO 'git'@'localhost'; + mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES ON `gitlabhq_production`.* TO 'git'@'localhost'; # Quit the database session mysql> \q diff --git a/doc/install/installation.md b/doc/install/installation.md index 52031e9b9a1..518b914fe67 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -115,8 +115,9 @@ Remove the old Ruby 1.8 if present Download Ruby and compile it: mkdir /tmp/ruby && cd /tmp/ruby - curl -L --progress http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.6.tar.gz | tar xz - cd ruby-2.1.6 + curl -O --progress https://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.7.tar.gz + echo 'e2e195a4a58133e3ad33b955c829bb536fa3c075 ruby-2.1.7.tar.gz' | shasum -c - && tar xzf ruby-2.1.7.tar.gz + cd ruby-2.1.7 ./configure --disable-install-rdoc make sudo make install @@ -131,11 +132,11 @@ Since GitLab 8.0, Git HTTP requests are handled by gitlab-git-http-server. This is a small daemon written in Go. To install gitlab-git-http-server we need a Go compiler. - curl -O --progress https://storage.googleapis.com/golang/go1.5.linux-amd64.tar.gz - echo '5817fa4b2252afdb02e11e8b9dc1d9173ef3bd5a go1.5.linux-amd64.tar.gz' | shasum -c - && \ - sudo tar -C /usr/local -xzf go1.5.linux-amd64.tar.gz + curl -O --progress https://storage.googleapis.com/golang/go1.5.1.linux-amd64.tar.gz + echo '46eecd290d8803887dec718c691cc243f2175fe0 go1.5.1.linux-amd64.tar.gz' | shasum -c - && \ + sudo tar -C /usr/local -xzf go1.5.1.linux-amd64.tar.gz sudo ln -sf /usr/local/go/bin/{go,godoc,gofmt} /usr/local/bin/ - rm go1.5.linux-amd64.tar.gz + rm go1.5.1.linux-amd64.tar.gz ## 4. System Users @@ -207,9 +208,9 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da ### Clone the Source # Clone GitLab repository - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 7-14-stable gitlab + sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 8-0-stable gitlab -**Note:** You can change `7-14-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! +**Note:** You can change `8-0-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! ### Configure It @@ -222,6 +223,10 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da # Update GitLab config file, follow the directions at top of file sudo -u git -H editor config/gitlab.yml + # Copy the example secrets file + sudo -u git -H cp config/secrets.yml.example config/secrets.yml + sudo -u git -H chmod 0600 config/secrets.yml + # Make sure GitLab can write to the log/ and tmp/ directories sudo chown -R git log/ sudo chown -R git tmp/ @@ -235,6 +240,9 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da # Make sure GitLab can write to the public/uploads/ directory sudo chmod -R u+rwX public/uploads + # Change the permissions of the directory where CI build traces are stored + sudo chmod -R u+rwX builds/ + # Copy the example Unicorn config sudo -u git -H cp config/unicorn.rb.example config/unicorn.rb @@ -299,7 +307,7 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da GitLab Shell is an SSH access and repository management software developed specially for GitLab. # Run the installation task for gitlab-shell (replace `REDIS_URL` if needed): - sudo -u git -H bundle exec rake gitlab:shell:install[v2.6.3] REDIS_URL=unix:/var/run/redis/redis.sock RAILS_ENV=production + sudo -u git -H bundle exec rake gitlab:shell:install[v2.6.5] REDIS_URL=unix:/var/run/redis/redis.sock RAILS_ENV=production # By default, the gitlab-shell config is generated from your main GitLab config. # You can review (and modify) the gitlab-shell config as follows: @@ -328,6 +336,17 @@ GitLab Shell is an SSH access and repository management software developed speci sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production GITLAB_ROOT_PASSWORD=yourpassword +### Secure secrets.yml + +The `secrets.yml` file stores encryption keys for sessions and secure variables. +Backup `secrets.yml` someplace safe, but don't store it in the same place as your database backups. +Otherwise your secrets are exposed if one of your backups is compromised. + +### Install schedules + + # Setup schedules + sudo -u gitlab_ci -H bundle exec whenever -w RAILS_ENV=production + ### Install Init Script Download the init script (will be `/etc/init.d/gitlab`): @@ -456,9 +475,22 @@ Using a self-signed certificate is discouraged but if you must use it follow the ``` 1. In the `config.yml` of gitlab-shell set `self_signed_cert` to `true`. -### Additional Markup Styles +### Enable Reply by email -Apart from the always supported markdown style there are other rich text files that GitLab can display. But you might have to install a dependency to do so. Please see the [github-markup gem readme](https://github.com/gitlabhq/markup#markups) for more information. +See the ["Reply by email" documentation](../incoming_email/README.md) for more information on how to set this up. + +### LDAP Authentication + +You can configure LDAP authentication in `config/gitlab.yml`. Please restart GitLab after editing this file. + +### Using Custom Omniauth Providers + +See the [omniauth integration document](../integration/omniauth.md) + +### Build your projects + +GitLab can build your projects. To enable that feature you need GitLab Runners to do that for you. +Checkout the [Gitlab Runner section](https://about.gitlab.com/gitlab-ci/#gitlab-runner) to install it ### Custom Redis Connection @@ -484,10 +516,16 @@ If you are running SSH on a non-standard port, you must change the GitLab user's You also need to change the corresponding options (e.g. `ssh_user`, `ssh_host`, `admin_uri`) in the `config\gitlab.yml` file. -### LDAP Authentication +### Additional Markup Styles -You can configure LDAP authentication in `config/gitlab.yml`. Please restart GitLab after editing this file. +Apart from the always supported markdown style there are other rich text files that GitLab can display. But you might have to install a dependency to do so. Please see the [github-markup gem readme](https://github.com/gitlabhq/markup#markups) for more information. -### Using Custom Omniauth Providers +## Troubleshooting -See the [omniauth integration document](../integration/omniauth.md) +### "You appear to have cloned an empty repository." + +If you see this message when attempting to clone a repository hosted by GitLab, +this is likely due to an outdated Nginx or Apache configuration, or a missing or +misconfigured `gitlab-git-http-server` instance. Double-check that you've +[installed Go](#3-go), [installed gitlab-git-http-server](#install-gitlab-git-http-server), +and correctly [configured Nginx](#site-configuration). diff --git a/doc/integration/README.md b/doc/integration/README.md index 6d856951d4e..eff39a626ae 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -10,7 +10,7 @@ See the documentation below for details on how to configure these services. - [SAML](saml.md) Configure GitLab as a SAML 2.0 Service Provider - [Slack](slack.md) Integrate with the Slack chat service - [OAuth2 provider](oauth_provider.md) OAuth2 application creation -- [Gmail](gitlab_buttons_in_gmail.md) Adds GitLab actions to messages +- [Gmail actions buttons](gmail_action_buttons_for_gitlab.md) Adds GitLab actions to messages GitLab Enterprise Edition contains [advanced JIRA support](http://doc.gitlab.com/ee/integration/jira.html) and [advanced Jenkins support](http://doc.gitlab.com/ee/integration/jenkins.html). diff --git a/doc/integration/gmail_action_buttons_for_gitlab.md b/doc/integration/gmail_action_buttons_for_gitlab.md new file mode 100644 index 00000000000..de45f25ad62 --- /dev/null +++ b/doc/integration/gmail_action_buttons_for_gitlab.md @@ -0,0 +1,22 @@ +# Gmail actions buttons for GitLab + +GitLab supports [Google actions in email](https://developers.google.com/gmail/markup/actions/actions-overview). + +If correctly setup, emails that require an action will be marked in Gmail. + + + +To get this functioning, you need to be registered with Google. +[See how to register with Google in this document.](https://developers.google.com/gmail/markup/registering-with-google) + +*This process has a lot of steps so make sure that you fulfill all requirements set by Google.* +*Your application will be rejected by Google if you fail to do so.* + +Pay close attention to: + +* Email account used by GitLab to send notification emails needs to have "Consistent history of sending a high volume of mail from your domain (order of hundred emails a day minimum to Gmail) for a few weeks at least". +* "A very very low rate of spam complaints from users." +* Emails must be authenticated via DKIM or SPF +* Before sending the final form("Gmail Schema Whitelist Request"), you must send a real email from your production server. This means that you will have to find a way to send this email from the email address you are registering. You can do this by, for example, forwarding the real email from the email address you are registering or going into the rails console on the GitLab server and triggering the email sending from there. + +You can check how it looks going through all the steps laid out in the "Registering with Google" doc in [this GitLab.com issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/1517). diff --git a/doc/integration/gmail_actions_button.png b/doc/integration/gmail_actions_button.png Binary files differnew file mode 100644 index 00000000000..b08f54d137b --- /dev/null +++ b/doc/integration/gmail_actions_button.png diff --git a/doc/integration/ldap.md b/doc/integration/ldap.md index 904d5d7fee2..9b7d8fa3969 100644 --- a/doc/integration/ldap.md +++ b/doc/integration/ldap.md @@ -78,6 +78,26 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server # user_filter: '' + # LDAP attributes that GitLab will use to create an account for the LDAP user. + # The specified attribute can either be the attribute name as a string (e.g. 'mail'), + # or an array of attribute names to try in order (e.g. ['mail', 'email']). + # Note that the user's LDAP login will always be the attribute specified as `uid` above. + attributes: + # The username will be used in paths for the user's own projects + # (like `gitlab.example.com/username/project`) and when mentioning + # them in issues, merge request and comments (like `@username`). + # If the attribute specified for `username` contains an email address, + # the GitLab username will be the part of the email address before the '@'. + username: ['uid', 'userid', 'sAMAccountName'] + email: ['mail', 'email', 'userPrincipalName'] + + # If no full name could be found at the attribute specified for `name`, + # the full name is determined using the attributes specified for + # `first_name` and `last_name`. + name: 'cn' + first_name: 'givenName' + last_name: 'sn' + # GitLab EE only: add more LDAP servers # Choose an ID made of a-z and 0-9 . This ID will be stored in the database # so that GitLab can remember which LDAP server a user belongs to. @@ -153,3 +173,23 @@ Tip: if you want to limit access to the nested members of an Active Directory gr ``` Please note that GitLab does not support the custom filter syntax used by omniauth-ldap. + +## Limitations + +GitLab's LDAP client is based on [omniauth-ldap](https://gitlab.com/gitlab-org/omniauth-ldap) +which encapsulates Ruby's `Net::LDAP` class. It provides a pure-Ruby implementation +of the LDAP client protocol. As a result, GitLab is limited by `omniauth-ldap` and may impact your LDAP +server settings. + +### TLS Client Authentication +Not implemented by `Net::LDAP`. +So you should disable anonymous LDAP authentication and enable simple or SASL +authentication. TLS client authentication setting in your LDAP server cannot be +mandatory and clients cannot be authenticated with the TLS protocol. + +### TLS Server Authentication +Not supported by GitLab's configuration options. +When setting `method: ssl`, the underlying authentication method used by +`omniauth-ldap` is `simple_tls`. This method establishes TLS encryption with +the LDAP server before any LDAP-protocol data is exchanged but no validation of +the LDAP server's SSL certificate is performed.
\ No newline at end of file diff --git a/doc/markdown/markdown.md b/doc/markdown/markdown.md index 322111ae9e1..ac3851f8c95 100644 --- a/doc/markdown/markdown.md +++ b/doc/markdown/markdown.md @@ -33,7 +33,6 @@ For GitLab we developed something we call "GitLab Flavored Markdown" (GFM). It e You can use GFM in -- commit messages - comments - issues - merge requests @@ -275,7 +274,7 @@ The IDs are generated from the content of the header according to the following 1. All spaces are converted to hyphens 1. Two or more hyphens in a row are converted to one 1. If a header with the same ID has already been generated, a unique - incrementing number is appended. + incrementing number is appended, starting at 1. For example: @@ -292,8 +291,8 @@ Would generate the following link IDs: 1. `this-header-has-spaces-in-it` 1. `this-header-has-a-in-it` 1. `this-header-has-unicode-in-it-한글` +1. `this-header-has-spaces-in-it` 1. `this-header-has-spaces-in-it-1` -1. `this-header-has-spaces-in-it-2` Note that the Emoji processing happens before the header IDs are generated, so the Emoji is converted to an image which then gets removed from the ID. @@ -413,7 +412,7 @@ Some text to show that the reference links can follow later. Relative links do not allow referencing project files in a wiki page or wiki page in a project file. The reason for this is that, in GitLab, wiki is always a separate git repository. For example: -`[I'm a reference-style link][style]` +`[I'm a reference-style link](style)` will point the link to `wikis/style` when the link is inside of a wiki markdown file. diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md new file mode 100644 index 00000000000..56bf7a14182 --- /dev/null +++ b/doc/migrate_ci_to_ce/README.md @@ -0,0 +1,321 @@ +## Migrate GitLab CI to GitLab CE or EE + +Beginning with version 8.0 of GitLab Community Edition (CE) and Enterprise +Edition (EE), GitLab CI is no longer its own application, but is instead built +into the CE and EE applications. + +This guide will detail the process of migrating your CI installation and data +into your GitLab CE or EE installation. **You can only migrate CI data from +GitLab CI 8.0 to GitLab 8.0; migrating between other versions (e.g.7.14 to 8.1) +is not possible.** + +We recommend that you read through the entire migration process in this +document before beginning. + +### Overview + +In this document we assume you have a GitLab server and a GitLab CI server. It +does not matter if these are the same machine. + +The migration consists of three parts: updating GitLab and GitLab CI, moving +data, and redirecting traffic. + +Please note that CI builds triggered on your GitLab server in the time between +updating to 8.0 and finishing the migration will be lost. Your GitLab server +can be online for most of the procedure; the only GitLab downtime (if any) is +during the upgrade to 8.0. Your CI service will be offline from the moment you +upgrade to 8.0 until you finish the migration procedure. + +### Before upgrading + +If you have GitLab CI installed using omnibus-gitlab packages but *you don't want to migrate your existing data*: + +```bash +mv /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds.$(date +%s) +``` + +and run `sudo gitlab-ctl reconfigure`. + +#### 1. Verify that backups work + +Make sure that the backup script on both servers can connect to the database. + +``` +# On your CI server: +# Omnibus +sudo gitlab-ci-rake backup:create + +# Source +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production +``` + +Also check on your GitLab server. + +``` +# On your GitLab server: +# Omnibus +sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads + +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads +``` + +If this fails you need to fix it before upgrading to 8.0. Also see +https://about.gitlab.com/getting-help/ + +#### 2. Check source and target database types + +Check what databases you use on your GitLab server and your CI server. + Look for the 'adapter:' line. If your CI server and your GitLab server use +the same database adapter no special care is needed. If your CI server uses +MySQL and your GitLab server uses PostgreSQL you need to pass a special option +during the 'Moving data' part. **If your CI server uses PostgreSQL and your +GitLab server uses MySQL you cannot migrate your CI data to GitLab 8.0.** + +``` +# On your CI server: +# Omnibus +sudo gitlab-ci-rake env:info + +# Source +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec rake env:info RAILS_ENV=production +``` + +``` +# On your GitLab server: +# Omnibus +sudo gitlab-rake gitlab:env:info + +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production +``` + +#### 3. Storage planning + +Decide where to store CI build traces on GitLab server. GitLab CI uses + files on disk to store CI build traces. The default path for these build +traces is `/var/opt/gitlab/gitlab-ci/builds` (Omnibus) or +`/home/git/gitlab/builds` (Source). If you are storing your repository data in +a special location, or if you are using NFS, you should make sure that you +store build traces on the same storage as your Git repositories. + +### I. Upgrading + +From this point on, GitLab CI will be unavailable for your end users. + +#### 1. Upgrade GitLab to 8.0 + +First upgrade your GitLab server to version 8.0: +https://about.gitlab.com/update/ + +#### 2. Disable CI on the GitLab server during the migration + +After you update, go to the admin panel and temporarily disable CI. As + an administrator, go to **Admin Area** -> **Settings**, and under +**Continuous Integration** uncheck **Disable to prevent CI usage until rake +ci:migrate is run (8.0 only)**. + +#### 3. CI settings are now in GitLab + +If you want to use custom CI settings (e.g. change where builds are + stored), please update `/etc/gitlab/gitlab.rb` (Omnibus) or +`/home/git/gitlab/config/gitlab.yml` (Source). + +#### 4. Upgrade GitLab CI to 8.0 + +Now upgrade GitLab CI to version 8.0. If you are using Omnibus packages, + this may have already happened when you upgraded GitLab to 8.0. + +#### 5. Disable GitLab CI on the CI server + +Disable GitLab CI after upgrading to 8.0. + +``` +# On your CI server: +# Omnibus +sudo gitlab-ctl stop ci-unicorn +sudo gitlab-ctl stop ci-sidekiq + +# Source +sudo service gitlab_ci stop +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec whenever --clear-crontab RAILS_ENV=production +``` + +### II. Moving data + +#### 1. Database encryption key + +Move the database encryption key from your CI server to your GitLab + server. The command below will show you what you need to copy-paste to your +GitLab server. On Omnibus GitLab servers you will have to add a line to +`/etc/gitlab/gitlab.rb`. On GitLab servers installed from source you will have +to replace the contents of `/home/git/gitlab/config/secrets.yml`. + +``` +# On your CI server: +# Omnibus +sudo gitlab-ci-rake backup:show_secrets + +# Source +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec rake backup:show_secrets RAILS_ENV=production +``` + +#### 2. SQL data and build traces + +Create your final CI data export. If you are converting from MySQL to + PostgreSQL, add ` MYSQL_TO_POSTGRESQL=1` to the end of the rake command. When +the command finishes it will print the path to your data export archive; you +will need this file later. + +``` +# On your CI server: +# Omnibus +sudo gitlab-ci-rake backup:create + +# Source +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production +``` + +#### 3. Copy data to the GitLab server + +If you were running GitLab and GitLab CI on the same server you can skip this +step. + +Copy your CI data archive to your GitLab server. There are many ways to do +this, below we use SSH agent forwarding and 'scp', which will be easy and fast +for most setups. You can also copy the data archive first from the CI server to +your laptop and then from your laptop to the GitLab server. + +``` +# Start from your laptop +ssh -A ci_admin@ci_server.example +# Now on the CI server +scp /path/to/12345_gitlab_ci_backup.tar gitlab_admin@gitlab_server.example:~ +``` + +#### 4. Move data to the GitLab backups folder + +Make the CI data archive discoverable for GitLab. We assume below that you +store backups in the default path, adjust the command if necessary. + +``` +# On your GitLab server: +# Omnibus +sudo mv /path/to/12345_gitlab_ci_backup.tar /var/opt/gitlab/backups/ + +# Source +sudo mv /path/to/12345_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups/ +``` + +#### 5. Import the CI data into GitLab. + +This step will delete any existing CI data on your GitLab server. There should +be no CI data yet because you turned CI on the GitLab server off earlier. + +``` +# On your GitLab server: +# Omnibus +sudo gitlab-rake ci:migrate + +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production +``` + +#### 6. Restart GitLab + +``` +# On your GitLab server: +# Omnibus +sudo gitlab-ctl hup unicorn +sudo gitlab-ctl restart sidekiq + +# Source +sudo service gitlab reload +``` + +### III. Redirecting traffic + +If you were running GitLab CI with Omnibus packages and you were using the +internal NGINX configuration your CI service should now be available both at +`ci.example.com` (the old address) and `gitlab.example.com/ci`. **You are done!** + +If you installed GitLab CI from source we now need to configure a redirect in +NGINX so that existing CI runners can keep using the old CI server address, and +so that existing links to your CI server keep working. + +#### 1. Update Nginx configuration + +To ensure that your existing CI runners are able to communicate with the +migrated installation, and that existing build triggers still work, you'll need +to update your Nginx configuration to redirect requests for the old locations to +the new ones. + +Edit `/etc/nginx/sites-available/gitlab_ci` and paste: + +```nginx +# GITLAB CI +server { + listen 80 default_server; # e.g., listen 192.168.1.1:80; + server_name YOUR_CI_SERVER_FQDN; # e.g., server_name source.example.com; + + access_log /var/log/nginx/gitlab_ci_access.log; + error_log /var/log/nginx/gitlab_ci_error.log; + + # expose API to fix runners + location /api { + proxy_read_timeout 300; + proxy_connect_timeout 300; + proxy_redirect off; + proxy_set_header X-Real-IP $remote_addr; + + # You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN + resolver 8.8.8.8 8.8.4.4; + proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; + } + + # redirect all other CI requests + location / { + return 301 $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; + } + + # adjust this to match the largest build log your runners might submit, + # set to 0 to disable limit + client_max_body_size 10m; +} +``` + +Make sure you substitute these placeholder values with your real ones: + +1. `YOUR_CI_SERVER_FQDN`: The existing public-facing address of your GitLab CI + install (e.g., `ci.gitlab.com`). +1. `YOUR_GITLAB_SERVER_FQDN`: The current public-facing address of your GitLab + CE (or EE) install (e.g., `gitlab.com`). + +**Make sure not to remove the `/ci$request_uri` part. This is required to +properly forward the requests.** + +You should also make sure that you can: + +1. `curl https://YOUR_GITLAB_SERVER_FQDN/` from your previous GitLab CI server. +1. `curl https://YOUR_CI_SERVER_FQDN/` from your GitLab CE (or EE) server. + +#### 2. Check Nginx configuration + + sudo nginx -t + +#### 3. Restart Nginx + + sudo /etc/init.d/nginx restart + +#### Restore from backup + +If something went wrong and you need to restore a backup, consult the [Backup +restoration](../raketasks/backup_restore.md) guide. diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index ad1914398d6..db3f6bb40bd 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -97,6 +97,8 @@ For installations from source: aws_secret_access_key: 'secret123' # The remote 'directory' to store your backups. For S3, this would be the bucket name. remote_directory: 'my.s3.bucket' + # Turns on AWS Server-Side Encryption with Amazon S3-Managed Keys for backups, this is optional + # encryption: 'AES256' ``` If you are uploading your backups to S3 you will probably want to create a new @@ -373,6 +375,8 @@ For more information see similar questions on postgresql issue tracker[here](htt This documentation is for GitLab CE. We backup GitLab.com and make sure your data is secure, but you can't use these methods to export / backup your data yourself from GitLab.com. +Issues are stored in the database. They can't be stored in Git itself. + To migrate your repositories from one server to another with an up-to-date version of GitLab, you can use the [import rake task](import.md) to do a mass import of the repository. Note that if you do an import rake task, rather than a backup restore, you diff --git a/doc/raketasks/cleanup.md b/doc/raketasks/cleanup.md index 96d67f7b5d6..8fbcbb983e9 100644 --- a/doc/raketasks/cleanup.md +++ b/doc/raketasks/cleanup.md @@ -12,7 +12,8 @@ sudo gitlab-rake gitlab:cleanup:dirs bundle exec rake gitlab:cleanup:dirs RAILS_ENV=production ``` -Remove repositories (global only for now) from `/home/git/repositories` if they don't exist in GitLab database. +Rename repositories from `/home/git/repositories` if they don't exist in GitLab database. +The repositories get a `+orphaned+TIMESTAMP` suffix so that they cannot block new repositories from being created. ``` # omnibus-gitlab diff --git a/doc/release/monthly.md b/doc/release/monthly.md index c1ed9e3b80e..c56e99a7005 100644 --- a/doc/release/monthly.md +++ b/doc/release/monthly.md @@ -195,7 +195,7 @@ This can happen before tagging because Omnibus uses tags in its own repo and SHA ## Update GitLab.com with the stable version - Deploy the package (should not need downtime because of the small difference with RC1) -- Deploy the package for ci.gitlab.com +- Deploy the package for gitlab.com/ci ## Release CE, EE and CI diff --git a/doc/reply_by_email/README.md b/doc/reply_by_email/README.md deleted file mode 100644 index 5d36f5121d1..00000000000 --- a/doc/reply_by_email/README.md +++ /dev/null @@ -1,180 +0,0 @@ -# Reply by email - -GitLab can be set up to allow users to comment on issues and merge requests by replying to notification emails. - -In order to do this, you need access to an IMAP-enabled email account, with a provider or server that supports [email sub-addressing](https://en.wikipedia.org/wiki/Email_address#Sub-addressing). Sub-addressing is a feature where any email to `user+some_arbitrary_tag@example.com` will end up in the mailbox for `user@example.com`, and is supported by providers such as Gmail, Yahoo! Mail, Outlook.com and iCloud, as well as the [Postfix](http://www.postfix.org/) mail server which you can run on-premises. - -## Set it up - -In this example, we'll use the Gmail address `gitlab-replies@gmail.com`. If you're actually using Gmail with Reply by email, make sure you have [IMAP access enabled](https://support.google.com/mail/troubleshooter/1668960?hl=en#ts=1665018) and [allow less secure apps to access the account](https://support.google.com/accounts/answer/6010255). - -### Installations from source - -1. Go to the GitLab installation directory: - - ```sh - cd /home/git/gitlab - ``` - -1. Find the `reply_by_email` section in `config/gitlab.yml`, enable the feature and enter the email address including a placeholder for the `reply_key`: - - ```sh - sudo editor config/gitlab.yml - ``` - - ```yaml - reply_by_email: - enabled: true - address: "gitlab-replies+%{reply_key}@gmail.com" - ``` - - As mentioned, the part after `+` is ignored, and this will end up in the mailbox for `gitlab-replies@gmail.com`. - -2. Find `config/mail_room.yml.example` and copy it to `config/mail_room.yml`: - - ```sh - sudo cp config/mail_room.yml.example config/mail_room.yml - ``` - -3. Uncomment the configuration options in `config/mail_room.yml` and fill in the details for your specific IMAP server and email account: - - ```sh - sudo editor config/mail_room.yml - ``` - - ```yaml - :mailboxes: - - - # IMAP server host - :host: "imap.gmail.com" - # IMAP server port - :port: 993 - # Whether the IMAP server uses SSL - :ssl: true - # Email account username. Usually the full email address. - :email: "gitlab-replies@gmail.com" - # Email account password - :password: "[REDACTED]" - # The name of the mailbox where incoming mail will end up. Usually "inbox". - :name: "inbox" - # Always "sidekiq". - :delivery_method: sidekiq - # Always true. - :delete_after_delivery: true - :delivery_options: - # The URL to the Redis server used by Sidekiq. Should match the URL in config/resque.yml. - :redis_url: redis://localhost:6379 - # Always "resque:gitlab". - :namespace: resque:gitlab - # Always "incoming_email". - :queue: incoming_email - # Always "EmailReceiverWorker" - :worker: EmailReceiverWorker - ``` - - -4. Find `lib/support/init.d/gitlab.default.example` and copy it to `/etc/default/gitlab`: - - ```sh - sudo cp lib/support/init.d/gitlab.default.example /etc/default/gitlab - ``` - -5. Edit `/etc/default/gitlab` to enable `mail_room`: - - ```sh - sudo editor /etc/default/gitlab - ``` - - ```sh - mail_room_enabled=true - ``` - -6. Restart GitLab: - - ```sh - sudo service gitlab restart - ``` - -7. Check if everything is configured correctly: - - ```sh - sudo bundle exec rake gitlab:reply_by_email:check RAILS_ENV=production - ``` - -8. Reply by email should now be working. - -### Omnibus package installations - -TODO - -### Development - -1. Go to the GitLab installation directory. - -1. Find the `reply_by_email` section in `config/gitlab.yml`, enable the feature and enter the email address including a placeholder for the `reply_key`: - - ```yaml - reply_by_email: - enabled: true - address: "gitlab-replies+%{reply_key}@gmail.com" - ``` - - As mentioned, the part after `+` is ignored, and this will end up in the mailbox for `gitlab-replies@gmail.com`. - -2. Find `config/mail_room.yml.example` and copy it to `config/mail_room.yml`: - - ```sh - sudo cp config/mail_room.yml.example config/mail_room.yml - ``` - -3. Uncomment the configuration options in `config/mail_room.yml` and fill in the details for your specific IMAP server and email account: - - ```yaml - :mailboxes: - - - # IMAP server host - :host: "imap.gmail.com" - # IMAP server port - :port: 993 - # Whether the IMAP server uses SSL - :ssl: true - # Email account username. Usually the full email address. - :email: "gitlab-replies@gmail.com" - # Email account password - :password: "[REDACTED]" - # The name of the mailbox where incoming mail will end up. Usually "inbox". - :name: "inbox" - # Always "sidekiq". - :delivery_method: sidekiq - # Always true. - :delete_after_delivery: true - :delivery_options: - # The URL to the Redis server used by Sidekiq. Should match the URL in config/resque.yml. - :redis_url: redis://localhost:6379 - # Always "resque:gitlab". - :namespace: resque:gitlab - # Always "incoming_email". - :queue: incoming_email - # Always "EmailReceiverWorker" - :worker: EmailReceiverWorker - ``` - -4. Uncomment the `mail_room` line in your `Procfile`: - - ```yaml - mail_room: bundle exec mail_room -q -c config/mail_room.yml - ``` - -6. Restart GitLab: - - ```sh - bundle exec foreman start - ``` - -7. Check if everything is configured correctly: - - ```sh - bundle exec rake gitlab:reply_by_email:check RAILS_ENV=development - ``` - -8. Reply by email should now be working. diff --git a/doc/ssh/README.md b/doc/ssh/README.md index 7cdcd11c04c..b6b8000af4e 100644 --- a/doc/ssh/README.md +++ b/doc/ssh/README.md @@ -3,27 +3,27 @@ ## SSH keys An SSH key allows you to establish a secure connection between your -computer and GitLab. - -Before generating an SSH key, check if your system already has one by -running `cat ~/.ssh/id_rsa.pub`. If you see a long string starting with -`ssh-rsa` or `ssh-dsa`, you can skip the ssh-keygen step. +computer and GitLab. Before generating an SSH key in your shell, check if your system +already has one by running the following command: +```bash +cat ~/.ssh/id_rsa.pub +``` -To generate a new SSH key, just open your terminal and use code below. The -ssh-keygen command prompts you for a location and filename to store the key -pair and for a password. When prompted for the location and filename, you -can press enter to use the default. +If you see a long string starting with `ssh-rsa` or `ssh-dsa`, you can skip the `ssh-keygen` step. -It is a best practice to use a password for an SSH key, but it is not +Note: It is a best practice to use a password for an SSH key, but it is not required and you can skip creating a password by pressing enter. Note that the password you choose here can't be altered or retrieved. +To generate a new SSH key, use the following command: ```bash ssh-keygen -t rsa -C "$your_email" ``` +This command will prompt you for a location and filename to store the key +pair and for a password. When prompted for the location and filename, you +can press enter to use the default. -Use the code below to show your public key. - +Use the command below to show your public key: ```bash cat ~/.ssh/id_rsa.pub ``` @@ -32,7 +32,7 @@ Copy-paste the key to the 'My SSH Keys' section under the 'SSH' tab in your user profile. Please copy the complete key starting with `ssh-` and ending with your username and host. -Use code below to copy your public key to the clipboard. Depending on your +To copy your public key to the clipboard, use code below. Depending on your OS you'll need to use a different command: **Windows:** @@ -72,6 +72,8 @@ access can happen through being a direct member of the project, or through a group. See `def accessible_deploy_keys` in `app/models/user.rb` for more information. +Deploy keys can be shared between projects, you just need to add them to each project. + ## Applications ### Eclipse diff --git a/doc/update/6.x-or-7.x-to-7.14.md b/doc/update/6.x-or-7.x-to-7.14.md index a1488474f60..b34fb12da6f 100644 --- a/doc/update/6.x-or-7.x-to-7.14.md +++ b/doc/update/6.x-or-7.x-to-7.14.md @@ -127,7 +127,7 @@ sudo apt-get install nodejs ```bash cd /home/git/gitlab-shell sudo -u git -H git fetch -sudo -u git -H git checkout v2.6.4 +sudo -u git -H git checkout v2.6.5 ``` ## 7. Install libs, migrations, etc. @@ -162,12 +162,12 @@ sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab TIP: to see what changed in `gitlab.yml.example` in this release use next command: ``` -git diff 6-0-stable:config/gitlab.yml.example 7.14-stable:config/gitlab.yml.example +git diff 6-0-stable:config/gitlab.yml.example 7-14-stable:config/gitlab.yml.example ``` * Make `/home/git/gitlab/config/gitlab.yml` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-14-stable/config/gitlab.yml.example but with your settings. * Make `/home/git/gitlab/config/unicorn.rb` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-14-stable/config/unicorn.rb.example but with your settings. -* Make `/home/git/gitlab-shell/config.yml` the same as https://gitlab.com/gitlab-org/gitlab-shell/blob/v2.6.0/config.yml.example but with your settings. +* Make `/home/git/gitlab-shell/config.yml` the same as https://gitlab.com/gitlab-org/gitlab-shell/blob/v2.6.5/config.yml.example but with your settings. * Copy rack attack middleware config ```bash diff --git a/doc/update/7.13-to-7.14.md b/doc/update/7.13-to-7.14.md index 7c2d3f4498a..6dd9727fb49 100644 --- a/doc/update/7.13-to-7.14.md +++ b/doc/update/7.13-to-7.14.md @@ -63,7 +63,7 @@ sudo -u git -H git checkout 7-14-stable-ee ```bash cd /home/git/gitlab-shell sudo -u git -H git fetch -sudo -u git -H git checkout v2.6.4 +sudo -u git -H git checkout v2.6.5 ``` ### 5. Install libs, migrations, etc. diff --git a/doc/update/7.14-to-8.0.md b/doc/update/7.14-to-8.0.md index 29a38d07b3d..552216be932 100644 --- a/doc/update/7.14-to-8.0.md +++ b/doc/update/7.14-to-8.0.md @@ -10,9 +10,9 @@ months after this vulnerability became known the GitLab installation guide still contained instructions that would install the outdated, 'vulnerable' Git version 2.1.2. -Run the following command to get your current Git version. +Run the following command to get your current Git version: -``` +```sh /usr/local/bin/git --version ``` @@ -63,39 +63,70 @@ sudo -u git -H git checkout 8-0-stable-ee ```bash cd /home/git/gitlab-shell sudo -u git -H git fetch -sudo -u git -H git checkout v2.6.4 +sudo -u git -H git checkout v2.6.5 ``` ### 5. Install gitlab-git-http-server -First we download Go 1.5 and install it into /usr/local/go. +First we download Go 1.5 and install it into `/usr/local/go`: + +```bash +curl -O --progress https://storage.googleapis.com/golang/go1.5.linux-amd64.tar.gz +echo '5817fa4b2252afdb02e11e8b9dc1d9173ef3bd5a go1.5.linux-amd64.tar.gz' | shasum -c - && \ + sudo tar -C /usr/local -xzf go1.5.linux-amd64.tar.gz +sudo ln -sf /usr/local/go/bin/{go,godoc,gofmt} /usr/local/bin/ +rm go1.5.linux-amd64.tar.gz +``` + +Now we download `gitlab-git-http-server` and install it in `/home/git/gitlab-git-http-server`: + +```bash +cd /home/git +sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-git-http-server.git +cd gitlab-git-http-server +sudo -u git -H make +``` + +Make sure your unicorn.rb file contains a 'listen' line for +'127.0.0.1:8080' and that this line is not commented out. + +``` +cd /home/git/gitlab +grep ^listen config/unicorn.rb - curl -O --progress https://storage.googleapis.com/golang/go1.5.linux-amd64.tar.gz - echo '5817fa4b2252afdb02e11e8b9dc1d9173ef3bd5a go1.5.linux-amd64.tar.gz' | shasum -c - && \ - sudo tar -C /usr/local -xzf go1.5.linux-amd64.tar.gz - sudo ln -sf /usr/local/go/bin/{go,godoc,gofmt} /usr/local/bin/ - rm go1.5.linux-amd64.tar.gz +# If there is no 'listen' line for 127.0.0.1:8080, add it: +sudo -u git tee -a config/unicorn.rb <<EOF +listen "127.0.0.1:8080", :tcp_nopush => true +EOF +``` -Now we download gitlab-git-http-server and install it in /home/git/gitlab-git-http-server. +If your Git repositories are in a directory other than `/home/git/repositories`, +you need to tell `gitlab-git-http-server` about it via `/etc/gitlab/default`. +See `lib/support/init.d/gitlab.default.example` for the options. - cd /home/git - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-git-http-server.git - cd gitlab-git-http-server - sudo -u git -H make +### 6. Copy secrets -If you put your Git repositories in a directory different from /home/git/repositories, you need to tell gitlab-git-http-server about it via /etc/gitlab/default. -See lib/support/init.d/gitlab.default.example for the options. +The `secrets.yml` file is used to store keys to encrypt sessions and encrypt secure variables. +When you run migrations make sure to store it someplace safe. +Don't store it in the same place as your database backups, +otherwise your secrets are exposed if one of your backups is compromised. -### 6. Install libs, migrations, etc. +``` +cd /home/git/gitlab +sudo -u git -H cp config/secrets.yml.example config/secrets.yml +sudo -u git -H chmod 0600 config/secrets.yml +``` + +### 7. Install libs, migrations, etc. ```bash cd /home/git/gitlab -# MySQL installations (note: the line below states '--without ... postgres') -sudo -u git -H bundle install --without development test postgres --deployment +# MySQL installations (note: the line below states '--without postgres') +sudo -u git -H bundle install --without postgres development test --deployment -# PostgreSQL installations (note: the line below states '--without ... mysql') -sudo -u git -H bundle install --without development test mysql --deployment +# PostgreSQL installations (note: the line below states '--without mysql') +sudo -u git -H bundle install --without mysql development test --deployment # Run database migrations sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production @@ -107,38 +138,55 @@ sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab ``` -### 7. Update config files +### 8. Update config files #### New configuration options for `gitlab.yml` -There are new configuration options available for [`gitlab.yml`](config/gitlab.yml.example). View them with the command below and apply them to your current `gitlab.yml`. +There are new configuration options available for [`gitlab.yml`](config/gitlab.yml.example). View them with the command below and apply them manually to your current `gitlab.yml`: -``` +```sh git diff origin/7-14-stable:config/gitlab.yml.example origin/8-0-stable:config/gitlab.yml.example -`````` +``` -#### New NGINX configuration +The new options include configuration of GitLab CI that are now being part of GitLab CE and EE. -Because of the new gitlab-git-http-server you need to update your NGINX configuration. -If you skip this step 'git clone' and 'git push' over HTTP(S) will stop working. +#### New Nginx configuration -``` -# Remove '-ssl' twice in the diff command below if you use HTTP instead of HTTPS +Because of the new `gitlab-git-http-server` you need to update your Nginx +configuration. If you skip this step 'git clone' and 'git push' over HTTP(S) +will stop working. + +View changes between the previous recommended Nginx configuration and the +current one: + +```sh +# For HTTPS configurations git diff origin/7-14-stable:lib/support/nginx/gitlab-ssl origin/8-0-stable:lib/support/nginx/gitlab-ssl + +# For HTTP configurations +git diff origin/7-14-stable:lib/support/nginx/gitlab origin/8-0-stable:lib/support/nginx/gitlab ``` -### 8. Start application +If you are using Apache instead of NGINX please see the updated [Apache templates](https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache). +Also note that because Apache does not support upstreams behind Unix sockets you will need to let gitlab-git-http-server listen on a TCP port. You can do this via [/etc/default/gitlab](https://gitlab.com/gitlab-org/gitlab-ce/blob/8-0-stable/lib/support/init.d/gitlab.default.example#L34). + +### 9. Migrate GitLab CI to GitLab CE/EE + +Now, GitLab CE and EE has CI integrated. However, migrations don't happen automatically and you need to do it manually. +Please follow the following guide [to migrate](../migrate_ci_to_ce/README.md) your GitLab CI instance to GitLab CE/EE. + +### 10. Start application sudo service gitlab start sudo service nginx restart -### 9. Check application status +### 11. Check application status Check if GitLab and its environment are configured correctly: sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production -To make sure you didn't miss anything run a more thorough check with: +To make sure you didn't miss anything run a more thorough check: sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production @@ -147,13 +195,25 @@ If all items are green, then congratulations, the upgrade is complete! ## Things went south? Revert to previous version (7.14) ### 1. Revert the code to the previous version + Follow the [upgrade guide from 7.13 to 7.14](7.13-to-7.14.md), except for the database migration (The backup is already migrated to the previous version) -### 2. Restore from the backup: +### 2. Restore from the backup ```bash cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production ``` -If you have more than one backup *.tar file(s) please add `BACKUP=timestamp_of_backup` to the command above. + +If you have more than one backup `*.tar` file(s) please add `BACKUP=timestamp_of_backup` to the command above. + +## Troubleshooting + +### "You appear to have cloned an empty repository." + +If you see this message when attempting to clone a repository hosted by GitLab, +this is likely due to an outdated Nginx or Apache configuration, or a missing or +misconfigured `gitlab-git-http-server` instance. Double-check that you correctly +completed [Step 5](#5-install-gitlab-git-http-server) to install the daemon and +[Step 8](#new-nginx-configuration) to reconfigure Nginx. diff --git a/doc/update/patch_versions.md b/doc/update/patch_versions.md index 22b9be059d6..a66a863f6c4 100644 --- a/doc/update/patch_versions.md +++ b/doc/update/patch_versions.md @@ -1,7 +1,7 @@ # Universal update guide for patch versions *Make sure you view this [upgrade guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/patch_versions.md) from the `master` branch for the most up to date instructions.* -For example from 6.2.0 to 6.2.1, also see the [semantic versioning specification](http://semver.org/). +For example from 7.14.0 to 7.14.3, also see the [semantic versioning specification](http://semver.org/). ### 0. Backup @@ -23,17 +23,16 @@ sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production cd /home/git/gitlab sudo -u git -H git fetch --all sudo -u git -H git checkout -- Gemfile.lock db/schema.rb -sudo -u git -H git checkout LATEST_TAG +LATEST_TAG=$(git describe --tags `git rev-list --tags --max-count=1`) +sudo -u git -H git checkout $LATEST_TAG -b $LATEST_TAG ``` -Replace LATEST_TAG with the latest GitLab tag you want to upgrade to, for example `v6.6.3`. - ### 3. Update gitlab-shell to the corresponding version ```bash cd /home/git/gitlab-shell sudo -u git -H git fetch -sudo -u git -H git checkout v`cat /home/git/gitlab/GITLAB_SHELL_VERSION` +sudo -u git -H git checkout v`cat /home/git/gitlab/GITLAB_SHELL_VERSION` -b v`cat /home/git/gitlab/GITLAB_SHELL_VERSION` ``` ### 4. Install libs, migrations, etc. diff --git a/doc/update/upgrader.md b/doc/update/upgrader.md index 6854250dab7..fd0327686b1 100644 --- a/doc/update/upgrader.md +++ b/doc/update/upgrader.md @@ -1,4 +1,4 @@ -# GitLab Upgrader +# GitLab Upgrader (deprecated) *DEPRECATED* We recommend to [switch to the Omnibus package and repository server](https://about.gitlab.com/update/) instead of using this script. diff --git a/doc/web_hooks/ssl.png b/doc/web_hooks/ssl.png Binary files differnew file mode 100644 index 00000000000..698f1a0f64a --- /dev/null +++ b/doc/web_hooks/ssl.png diff --git a/doc/web_hooks/web_hooks.md b/doc/web_hooks/web_hooks.md index 09400d9b163..c185ccfcac3 100644 --- a/doc/web_hooks/web_hooks.md +++ b/doc/web_hooks/web_hooks.md @@ -6,7 +6,15 @@ You can configure web hooks to listen for specific events like pushes, issues or Web hooks can be used to update an external issue tracker, trigger CI builds, update a backup mirror, or even deploy to your production server. -If you send a web hook to an SSL endpoint [the certificate will not be verified](https://gitlab.com/gitlab-org/gitlab-ce/blob/ccd617e58ea71c42b6b073e692447d0fe3c00be6/app/models/web_hook.rb#L35) since many people use self-signed certificates. +## SSL Verification + +By default, the SSL certificate of the webhook endpoint is verified based on +an internal list of Certificate Authorities, +which means the certificate cannot be self-signed. + +You can turn this off in the web hook settings in your GitLab projects. + + ## Push events @@ -34,7 +42,7 @@ X-Gitlab-Event: Push Hook "name": "Diaspora", "url": "git@example.com:mike/diasporadiaspora.git", "description": "", - "homepage": "http://example.com/mike/diaspora", + "homepage": "http://example.com/mike/diaspora", "git_http_url":"http://example.com/mike/diaspora.git", "git_ssh_url":"git@example.com:mike/diaspora.git", "visibility_level":0 @@ -121,6 +129,12 @@ X-Gitlab-Event: Issue Hook "username": "root", "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" }, + "repository": { + "name": "Gitlab Test", + "url": "http://example.com/gitlabhq/gitlab-test.git", + "description": "Aut reprehenderit ut est.", + "homepage": "http://example.com/gitlabhq/gitlab-test" + }, "object_attributes": { "id": 301, "title": "New API: create/update/delete file", @@ -507,8 +521,8 @@ server.mount_proc '/' do |req, res| puts req.body end -trap 'INT' do - server.shutdown +trap 'INT' do + server.shutdown end server.start ``` @@ -523,4 +537,4 @@ When you press 'Test Hook' in GitLab, you should see something like this in the {"before":"077a85dd266e6f3573ef7e9ef8ce3343ad659c4e","after":"95cd4a99e93bc4bbabacfa2cd10e6725b1403c60",<SNIP>} example.com - - [14/May/2014:07:45:26 EDT] "POST / HTTP/1.1" 200 0 - -> / -``` +```
\ No newline at end of file diff --git a/doc/workflow/importing/README.md b/doc/workflow/importing/README.md index 5cde90993d2..7ccf06fbd60 100644 --- a/doc/workflow/importing/README.md +++ b/doc/workflow/importing/README.md @@ -3,6 +3,7 @@ 1. [Bitbucket](import_projects_from_bitbucket.md)
2. [GitHub](import_projects_from_github.md)
3. [GitLab.com](import_projects_from_gitlab_com.md)
+4. [FogBugz](import_projects_from_fogbugz.md)
4. [SVN](migrating_from_svn.md)
### Note
diff --git a/doc/workflow/importing/fogbugz_importer/fogbugz_import_finished.png b/doc/workflow/importing/fogbugz_importer/fogbugz_import_finished.png Binary files differnew file mode 100644 index 00000000000..205c515bd3f --- /dev/null +++ b/doc/workflow/importing/fogbugz_importer/fogbugz_import_finished.png diff --git a/doc/workflow/importing/fogbugz_importer/fogbugz_import_login.png b/doc/workflow/importing/fogbugz_importer/fogbugz_import_login.png Binary files differnew file mode 100644 index 00000000000..a1e348d46ad --- /dev/null +++ b/doc/workflow/importing/fogbugz_importer/fogbugz_import_login.png diff --git a/doc/workflow/importing/fogbugz_importer/fogbugz_import_select_fogbogz.png b/doc/workflow/importing/fogbugz_importer/fogbugz_import_select_fogbogz.png Binary files differnew file mode 100644 index 00000000000..ed362846909 --- /dev/null +++ b/doc/workflow/importing/fogbugz_importer/fogbugz_import_select_fogbogz.png diff --git a/doc/workflow/importing/fogbugz_importer/fogbugz_import_select_project.png b/doc/workflow/importing/fogbugz_importer/fogbugz_import_select_project.png Binary files differnew file mode 100644 index 00000000000..d2fbd0267bd --- /dev/null +++ b/doc/workflow/importing/fogbugz_importer/fogbugz_import_select_project.png diff --git a/doc/workflow/importing/fogbugz_importer/fogbugz_import_user_map.png b/doc/workflow/importing/fogbugz_importer/fogbugz_import_user_map.png Binary files differnew file mode 100644 index 00000000000..b1cc4b58525 --- /dev/null +++ b/doc/workflow/importing/fogbugz_importer/fogbugz_import_user_map.png diff --git a/doc/workflow/importing/import_projects_from_fogbugz.md b/doc/workflow/importing/import_projects_from_fogbugz.md new file mode 100644 index 00000000000..71af0f9ea44 --- /dev/null +++ b/doc/workflow/importing/import_projects_from_fogbugz.md @@ -0,0 +1,29 @@ +# Import your project from FogBugz to GitLab + +It only takes a few simple steps to import your project from FogBugz. +The importer will import all of your cases and comments with original case +numbers and timestamps. You will also have the opportunity to map FogBugz +users to GitLab users. + +* From your GitLab dashboard click 'New project' + +* Click on the 'FogBugz' button + + + +* Enter your FogBugz URL, email address, and password. + + + +* Create mapping from FogBugz users to GitLab users. + + + +* Select the projects you wish to import by clicking the Import buttons + + + +* Once the import has finished click the link to take you to the project +dashboard. Follow the directions to push your existing repository. + + |
