diff options
| -rw-r--r-- | .gitlab-ci.yml | 24 | ||||
| -rw-r--r-- | doc/development/automatic_ce_ee_merge.md | 283 | ||||
| -rwxr-xr-x | scripts/merge-train | 73 |
3 files changed, 203 insertions, 177 deletions
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index caec485d845..4e71140fbc9 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -48,6 +48,7 @@ after_script: stages: - build - prepare + - merge - test - post-test - pages @@ -1025,3 +1026,26 @@ schedule:review-cleanup: - gem install gitlab --no-document script: - ruby -rrubygems scripts/review_apps/automated_cleanup.rb + +merge:master: + image: registry.gitlab.com/gitlab-org/merge-train + stage: merge + # The global before_script/after_script blocks break this job, or aren't + # necessary. These two lines result in them being ignored. + before_script: [] + after_script: [] + only: + refs: + - master + - schedules + variables: + - $CI_PROJECT_PATH == "gitlab-org/gitlab-ce" + - $MERGE_TRAIN_SSH_PUBLIC_KEY + - $MERGE_TRAIN_SSH_PRIVATE_KEY + - $MERGE_TRAIN_API_TOKEN + script: + - scripts/merge-train + cache: + paths: + - gitlab-ee + key: "merge:master" diff --git a/doc/development/automatic_ce_ee_merge.md b/doc/development/automatic_ce_ee_merge.md index 58e08d432cc..e4eb26b3aca 100644 --- a/doc/development/automatic_ce_ee_merge.md +++ b/doc/development/automatic_ce_ee_merge.md @@ -1,57 +1,33 @@ # Automatic CE->EE merge -GitLab Community Edition is merged automatically every 3 hours into the -Enterprise Edition (look for the [`CE Upstream` merge requests]). - -This merge is done automatically in a -[scheduled pipeline](https://gitlab.com/gitlab-org/release-tools/-/jobs/43201679). - -## What to do if you are pinged in a `CE Upstream` merge request to resolve a conflict? - -1. Please resolve the conflict as soon as possible or ask someone else to do it - - It's ok to resolve more conflicts than the one that you are asked to resolve. - In that case, it's a good habit to ask for a double-check on your resolution - by someone who is familiar with the code you touched. -1. Once you have resolved your conflicts, push to the branch (no force-push) -1. Assign the merge request to the next person that has to resolve a conflict -1. If all conflicts are resolved after your resolution is pushed, keep the merge - request assigned to you: **you are now responsible for the merge request to be - green** -1. If you are the last person to resolve the conflicts, the pipeline is green, - and you have merge rights, merge the MR, but **do not** choose to squash. - Otherwise, assign the MR to someone that can merge. -1. If you need any help, you can ping the current [release managers], or ask in - the `#ce-to-ee` Slack channel - -A few notes about the automatic CE->EE merge job: - -- If a merge is already in progress, the job - [doesn't create a new one](https://gitlab.com/gitlab-org/release-tools/-/jobs/43157687). -- If there is nothing to merge (i.e. EE is up-to-date with CE), the job doesn't - create a new one -- The job posts messages to the `#ce-to-ee` Slack channel to inform what's the - current CE->EE merge status (e.g. "A new MR has been created", "A MR is still pending") - -[`CE Upstream` merge requests]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests?label_name%5B%5D=CE+upstream -[release managers]: https://about.gitlab.com/release-managers/ +Whenever a commit is pushed to the CE `master` branch, it is automatically +merged into the EE `master` branch. If the commit produces any conflicts, it is +instead reverted from CE `master`. When this happens, a merge request will be +set up automatically that can be used to reinstate the changes. This merge +request will be assigned to the author of the conflicting commit, or the merge +request author if the commit author could not be associated with a GitLab user. +If no author could be found, the merge request is assigned to a random member of +the Delivery team. It is then up to this team member to figure out who to assign +the merge request to. + +Because some commits can not be reverted if new commits depend on them, we also +run a job periodically that processes a range of commits and tries to merge or +revert them. This should ensure that all commits are either merged into EE +`master`, or reverted, instead of just being left behind in CE. ## Always merge EE merge requests before their CE counterparts **In order to avoid conflicts in the CE->EE merge, you should always merge the EE version of your CE merge request first, if present.** -The rationale for this is that as CE->EE merges are done automatically every few -hours, it can happen that: +The rationale for this is that as CE->EE merges are done automatically, it can +happen that: -1. A CE merge request that needs EE-specific changes is merged -1. The automatic CE->EE merge happens +1. A CE merge request that needs EE-specific changes is merged. +1. The automatic CE->EE merge happens. 1. Conflicts due to the CE merge request occur since its EE merge request isn't - merged yet -1. The automatic merge bot will ping someone to resolve the conflict **that are - already resolved in the EE merge request that isn't merged yet** - -That's a waste of time, and that's why you should merge EE merge request before -their CE counterpart. + merged yet. +1. The CE changes are reverted. ## Avoiding CE->EE merge conflicts beforehand @@ -69,136 +45,89 @@ detect if the current branch's changes will conflict during the CE->EE merge. The job reports what files are conflicting and how to set up a merge request against EE. -#### How the job works - -1. Generates the diff between your branch and current CE `master` -1. Tries to apply it to current EE `master` -1. If it applies cleanly, the job succeeds, otherwise... -1. Detects a branch with the `ee-` prefix or `-ee` suffix in EE -1. If it exists, generate the diff between this branch and current EE `master` -1. Tries to apply it to current EE `master` -1. If it applies cleanly, the job succeeds - -In the case where the job fails, it means you should create an `ee-<ce_branch>` -or `<ce_branch>-ee` branch, push it to EE and open a merge request against EE -`master`. -At this point if you retry the failing job in your CE merge request, it should -now pass. - -Notes: - -- This task is not a silver-bullet, its current goal is to bring awareness to - developers that their work needs to be ported to EE. -- Community contributors shouldn't be required to submit merge requests against - EE, but reviewers should take actions by either creating such EE merge request - or asking a GitLab developer to do it **before the merge request is merged**. -- If you branch is too far behind `master`, the job will fail. In that case you - should rebase your branch upon latest `master`. -- Code reviews for merge requests often consist of multiple iterations of - feedback and fixes. There is no need to update your EE MR after each - iteration. Instead, create an EE MR as soon as you see the - `ee_compat_check` job failing. After you receive the final approval - from a Maintainer (but **before the CE MR is merged**) update the EE MR. - This helps to identify significant conflicts sooner, but also reduces the - number of times you have to resolve conflicts. -- Please remember to - [always have your EE merge request merged before the CE version](#always-merge-ee-merge-requests-before-their-ce-counterparts). -- You can use [`git rerere`](https://git-scm.com/docs/git-rerere) - to avoid resolving the same conflicts multiple times. - -### Cherry-picking from CE to EE - -For avoiding merge conflicts, we use a method of creating equivalent branches -for CE and EE. If the `ee-compat-check` job fails, this process is required. - -This method only requires that you have cloned both CE and EE into your computer. -If you don't have them yet, please go ahead and clone them: - -- Clone CE repo: `git clone git@gitlab.com:gitlab-org/gitlab-ce.git` -- Clone EE repo: `git clone git@gitlab.com:gitlab-org/gitlab-ee.git` - -And the only additional setup we need is to add CE as remote of EE and vice-versa: - -- Open two terminal windows, one in CE, and another one in EE: - - In EE: `git remote add ce git@gitlab.com:gitlab-org/gitlab-ce.git` - - In CE: `git remote add ee git@gitlab.com:gitlab-org/gitlab-ee.git` - -That's all setup we need, so that we can cherry-pick a commit from CE to EE, and -from EE to CE. - -Now, every time you create an MR for CE and EE: - -1. Open two terminal windows, one in CE, and another one in EE -1. In the CE terminal: - 1. Create the CE branch, e.g., `branch-example` - 1. Make your changes and push a commit (commit A) - 1. Create the CE merge request in GitLab -1. In the EE terminal: - 1. Create the EE-equivalent branch ending with `-ee`, e.g., - `git checkout -b branch-example-ee` - 1. Fetch the CE branch: `git fetch ce branch-example` - 1. Cherry-pick the commit A: `git cherry-pick commit-A-SHA` - 1. If Git prompts you to fix the conflicts, do a `git status` - to check which files contain conflicts, fix them, save the files - 1. Add the changes with `git add .` but **DO NOT commit** them - 1. Continue cherry-picking: `git cherry-pick --continue` - 1. Push to EE: `git push origin branch-example-ee` -1. Create the EE-equivalent MR and link to the CE MR from the -description "Ports [CE-MR-LINK] to EE" -1. Once all the jobs are passing in both CE and EE, you've addressed the -feedback from your own team, and got them approved, the merge requests can be merged. -1. When both MRs are ready, the EE merge request will be merged first, and the -CE-equivalent will be merged next. - -**Important notes:** - -- The commit SHA can be easily found from the GitLab UI. From a merge request, -open the tab **Commits** and click the copy icon to copy the commit SHA. -- To cherry-pick a **commit range**, such as [A > B > C > D] use: - - ```shell - git cherry-pick "oldest-commit-SHA^..newest-commit-SHA" - ``` - - For example, suppose the commit A is the oldest, and its SHA is `4f5e4018c09ed797fdf446b3752f82e46f5af502`, - and the commit D is the newest, and its SHA is `80e1c9e56783bd57bd7129828ec20b252ebc0538`. - The cherry-pick command will be: - - ```shell - git cherry-pick "4f5e4018c09ed797fdf446b3752f82e46f5af502^..80e1c9e56783bd57bd7129828ec20b252ebc0538" - ``` - -- To cherry-pick a **merge commit**, use the flag `-m 1`. For example, suppose that the -merge commit SHA is `138f5e2f20289bb376caffa0303adb0cac859ce1`: - - ```shell - git cherry-pick -m 1 138f5e2f20289bb376caffa0303adb0cac859ce1 - ``` -- To cherry-pick multiple commits, such as B and D in a range [A > B > C > D], use: - - ```shell - git cherry-pick commmit-B-SHA commit-D-SHA - ``` - - For example, suppose commit B SHA = `4f5e4018c09ed797fdf446b3752f82e46f5af502`, - and the commit D SHA = `80e1c9e56783bd57bd7129828ec20b252ebc0538`. - The cherry-pick command will be: - - ```shell - git cherry-pick 4f5e4018c09ed797fdf446b3752f82e46f5af502 80e1c9e56783bd57bd7129828ec20b252ebc0538 - ``` - - This case is particularly useful when you have a merge commit in a sequence of - commits and you want to cherry-pick all but the merge commit. - -- If you push more commits to the CE branch, you can safely repeat the procedure -to cherry-pick them to the EE-equivalent branch. You can do that as many times as -necessary, using the same CE and EE branches. -- If you submitted the merge request to the CE repo and the `ee-compat-check` job passed, -you are not required to submit the EE-equivalent MR, but it's still recommended. If the -job failed, you are required to submit the EE MR so that you can fix the conflicts in EE -before merging your changes into CE. - ---- - -[Return to Development documentation](README.md) +## How to reinstate changes + +When a commit is reverted, the corresponding merge request to reinstate the +changes will include all the details necessary to ensure the changes make it +back into CE and EE. However, you still need to manually set up an EE merge +request that resolves the conflicts. + +Each merge request used to reinstate changes will have the "reverted" label +applied. Please do not remove this label, as it will be used to determine how +many times commits are reverted and how long it takes to reinstate the changes. + +An example merge request can be found in [CE merge request +23280](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23280). + +## How it works + +The automatic merging is performed using a project called [Merge +Train](https://gitlab.com/gitlab-org/merge-train/). For every commit to merge or +revert, we generate patches using `git format-patch` which we then try to apply +using `git am --3way`. If this succeeds we push the changes to EE, if this fails +we decide what to do based on the failure reason: + +1. If the patch could not be applied because it was already applied, we just + skip it. +1. If the patch caused conflicts, we revert the source commits. + +Commits are reverted in reverse order, ensuring that if commit B depends on A, +and both conflict, we first revert B followed by reverting A. + +## FAQ + +### Why? + +We want to work towards being able to deploy continuously, but this requires +that `master` is always stable and has all the changes we need. If CE `master` +can not be merged into EE `master` due to merge conflicts, this prevents _any_ +change from CE making its way into EE. Since GitLab.com runs on EE, this +effectively prevents us from deploying changes. + +Past experiences and data have shown that periodic CE to EE merge requests do +not scale, and often take a very long time to complete. For example, [in this +comment](https://gitlab.com/gitlab-org/release/framework/issues/49#note_114614619) +we determined that the average time to close an upstream merge request is around +5 hours, with peaks up to several days. Periodic merge requests are also +frustrating to work with, because they often include many changes unrelated to +your own changes. + +Automatically merging or reverting commits allows us to keep merging changes +from CE into EE, as we never have to wait hours for somebody to resolve a set of +merge conflicts. + +### Does the CE to EE merge take into account merge commits? + +No. When merging CE changes into EE, merge commits are ignored. + +### My changes are reverted, but I set up an EE MR to resolve conflicts + +Most likely the automatic merge job ran before the EE merge request was merged. +If this keeps happening, consider reporting a bug in the [Merge Train issue +tracker](https://gitlab.com/gitlab-org/merge-train/issues). + +### My changes keep getting reverted, and this is really annoying! + +This is understandable, but the solution to this is fairly straightforward: +simply set up an EE merge request for every CE merge request, and resolve your +conflicts before the changes are reverted. + +### Will we allow certain people to still merge changes, even if they conflict? + +No. + +### Some files I work with often conflict, how can I best deal with this? + +If you find you keep running into merge conflicts, consider refactoring the file +so that the EE specific changes are not intertwined with CE code. For Ruby code +you can do this by moving the EE code to a separate module, which can then be +injected into the appropriate classes or modules. See [Guidelines for +implementing Enterprise Edition features](ee_features.md) for more information. + +### Will changelog entries be reverted automatically? + +Only if the changelog was added in the commit that was reverted. If a changelog +entry was added in a separate commit, it is possible for it to be left behind. +Since changelog entries are related to the changes in question, there is no real +reason to commit the changelog separately, and as such this should not be a big +problem. diff --git a/scripts/merge-train b/scripts/merge-train new file mode 100755 index 00000000000..a934971b869 --- /dev/null +++ b/scripts/merge-train @@ -0,0 +1,73 @@ +#!/bin/sh + +set -e + +# The name (including namespace) of the EE repository to merge commits into. +EE_PROJECT='gitlab-org/gitlab-ee' + +# The directory to clone GitLab EE into. +EE_DIRECTORY="$CI_PROJECT_DIR/gitlab-ee" + +# The EE branch to merge the changes into. +EE_BRANCH='master' + +# Runs an incremental or periodic merge of CE to EE. This script should be run +# from a container built using https://gitlab.com/gitlab-org/merge-train. + +# Merges (or reverts) commits in a batch (based on CI_COMMIT_BEFORE_SHA and +# CI_COMMIT_SHA), or since a specific commit. +# +# The optional first argument of this function should be a SHA of a commit. When +# specified, all commits since this commit will be processed. +merge() { + # We need to source the configure-ssh script instead of running it with + # `sh`, since it uses `eval` for SSH agent and we want the result of that to + # persist. + # + # shellcheck disable=SC1091 + . /app/bin/configure-ssh + + # We can not perform a shallow clone, as this results in Git sometimes + # refusing to fetch from CE. To work around this, we perform a full clone + # but cache the repository in CI. If a cached repository exists, we simply + # just pull from `master`. + if [ -d "$EE_DIRECTORY" ] + then + echo "Updating existing clone of $EE_PROJECT" + + git -C "$EE_DIRECTORY" pull --quiet origin "$EE_BRANCH" + else + echo "Cloning $EE_PROJECT" + + git clone --quiet --single-branch --branch "$EE_BRANCH" \ + "git@gitlab.com:$EE_PROJECT.git" "$EE_DIRECTORY" + fi + + cd /app + + env GITLAB_TOKEN="$MERGE_TRAIN_API_TOKEN" \ + bundle exec /app/bin/merge-train "$CI_PROJECT_DIR" "$EE_DIRECTORY" \ + --source-name "$CI_PROJECT_PATH" \ + --target-branch "$EE_BRANCH" \ + ${1:+--before "$1"} +} + +# Merges (or reverts) all commits since a point in time as supported by `git log +# --since`, such as "12 hours ago". +merge_since() { + commit="$(git log --since="$MERGE_SINCE" --reverse --format=%H | head -n1)" + + if [ "$commit" = '' ] + then + echo "There are no commits to merge since $MERGE_SINCE" + else + merge "$commit" + fi +} + +if [ "$MERGE_SINCE" ] +then + merge_since +else + merge +fi |