summaryrefslogtreecommitdiff
path: root/lib/gitlab/background_migration
Commit message (Collapse)AuthorAgeFilesLines
* Handle null serialised commits in background migration39181-gitlab-backgroundmigration-deserializemergerequestdiffsandcommits-error-nomethoderror-undefined-method-map-for-nil-nilclassSean McGivern2017-10-171-0/+1
| | | | This is already handled for diffs, but not commits.
* Don't create fork networks for root projects that are deletedbvl-fork-network-migrationsBob Van Landuyt2017-10-092-0/+10
|
* Sync up hard coded DN class in migrationmk-normalize-ldap-user-dnsMichael Kozono2017-10-071-20/+28
|
* Refactor DN error classesMichael Kozono2017-10-071-14/+15
|
* Move migration to backgroundMichael Kozono2017-10-071-0/+304
|
* Add a migration to populate fork networksBob Van Landuyt2017-10-072-0/+114
| | | | This uses the existing ForkedProjectLinks
* Don't call update signature worker when there aren't subkeysRubén Dávila2017-10-051-1/+4
|
* Convert migrations to generate subkeys to a background migrationRubén Dávila2017-10-051-0/+50
|
* Make MR diff background migration less likely to time outSean McGivern2017-09-291-9/+25
| | | | | | | | | | | | | | | This version does not use transactions, but individual statements. As we have unique constraints on the target tables for the inserts, we can just ignore uniqueness violations there (as long as we always insert the same batch size, in the same order). This means the spec now must use truncation, not a transaction, as the uniqueness violation means that the whole transaction for that spec would be invalid, which isn't what we'd want. In real-world use, this isn't run in a transaction anyway. This commit also wraps unhandled exceptions, for easier finding in Sentry, and logs with a consistent format, for easier searching.
* Insert at most 1,000 rows at once in MR diff background migrationSean McGivern2017-09-291-3/+9
| | | | | | We were hitting the statement timeout for very large MR diffs. Now we insert at most 1,000 rows to `merge_request_diff_commits` in a single statement, or 100 rows to `merge_request_diff_files`.
* Simplify queryMichael Kozono2017-09-141-15/+4
| | | | Performance does not need to be optimized in this case because this will be run in batches at a controlled rate.
* Add specs for deleting conflicting redirectsMichael Kozono2017-09-141-2/+2
|
* Delete conflicting redirects in backgroundMichael Kozono2017-09-141-0/+52
|
* Merge branch 'backstage/gb/migrate-stages-statuses' into 'master'Kamil Trzciński2017-08-211-0/+77
|\ | | | | | | | | | | | | Migrate CI/CD stages statuses Closes #33453 See merge request !12584
| * Merge branch 'master' into backstage/gb/migrate-stages-statusesGrzegorz Bizon2017-08-163-0/+364
| |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * master: (1000 commits) Fix username autocomplete group name with no avatar alignment Fix 'Projected tags' typo in protected_tags_spec.rb Many Repo Fixes Repo Editor Fixes Docs: New index for permissions link article from CI index link tech articles from the landing page new articles come first fix relative link fix date format Fixed changed files dropdown not being shown Update publication date Remove deprecated field from workhorse API responses Fix API responses when dealing with txt files Make sure MySQL would not use CURRENT_TIMESTAMP Add two more project templates Allow usage of any_projects? with an Array Copyedit Artifactory and GitLab article Rename Artifactory and GitLab article file Display GPG status loading spinner only when Ajax request is made ...
| * \ Merge branch 'master' into backstage/gb/migrate-stages-statusesGrzegorz Bizon2017-07-201-0/+26
| |\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * master: (319 commits) remove redundant changelog entries Merge branch '24570-use-re2-for-user-supplied-regexp-9-3' into 'security-9-3' Merge branch '33303-404-for-unauthorized-project' into 'security-9-3' Merge branch '33359-pers-snippet-files-location' into 'security-9-3' Merge branch 'bvl-remove-appearance-symlink' into 'security-9-3' Hide description about protected branches to non-member Update CHANGELOG.md for 9.0.11 Update CHANGELOG.md for 9.1.8 Update CHANGELOG.md for 8.17.7 Update CHANGELOG.md for 9.2.8 Update CHANGELOG.md for 9.3.8 Respect blockquote line breaks in markdown 35209 Add wip message to new navigation preference section Add github imported projects count to usage data Add versions to Prometheus metrics doc Add Bulgarian translations of Pipeline Schedules Add Esperanto translations of Pipeline Schedules Add Traditional Chinese in HongKong translations of Pipeline Schedules Add Simplified Chinese translations of Pipeline Schedules Resolve "Clarify k8s service keys" ... Conflicts: db/schema.rb
| * | | Freeze mutable constants in stages migration codeGrzegorz Bizon2017-07-111-1/+1
| | | |
| * | | Schedule stages statuses bg migrations in batchesGrzegorz Bizon2017-07-101-2/+3
| | | |
| * | | Merge branch 'master' into backstage/gb/migrate-stages-statusesGrzegorz Bizon2017-07-101-5/+5
| |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * master: (587 commits) Increase GitLab QA screenshots size Fix QA test scenario for creating a new project Configure headless chrome screenshots correctly Allow passing exceptions when creating project services Also inject new route helpers into includers of GitlabRoutingHelper Remove many N+1 queries with merge requests API fix milestones finder failing spec Make wrong migrations idempotent Remove double border on last group row Update tests to test what the component renders and what the clicks calls Username and password are no longer stripped from import url on import Change `sign_out` usage to `gitlab_sign_out` in one spec Send Gitaly Repository with /api/internal/allowed Resolve "Liberation Mono weird font rendering on Fedora/openSUSE/other Linux distros" Force mobile view for admin projects fix spec Mark a subgroup-related spec as PostgreSQL-only Use vue .prevent to prevent the click Add members-list class to admin projects page Fix another typo in SHA attribute spec ... Conflicts: db/post_migrate/20170628080858_migrate_stage_id_reference_in_background.rb db/schema.rb lib/gitlab/background_migration/migrate_build_stage_id_reference.rb spec/migrations/migrate_stage_id_reference_in_background_spec.rb
| * | | | Move stages status migration to the background workerbackstage/gb/migrate-stages-statusesGrzegorz Bizon2017-07-051-0/+76
| | | | |
| * | | | Sanitize id value passed to async background migrationGrzegorz Bizon2017-06-301-3/+1
| | | | |
| * | | | Improve readability of build stage id migration queryGrzegorz Bizon2017-06-291-6/+7
| | | | |
| * | | | Test if argument passed to a migration is presentGrzegorz Bizon2017-06-291-1/+1
| | | | |
| * | | | Add initial build stage_id ref background migrationGrzegorz Bizon2017-06-281-7/+11
| | | | |
| * | | | Add initial stage_id background migration filesGrzegorz Bizon2017-06-281-0/+16
| | | | |
* | | | | Fix MR diffs background migration when file modes are missing36630-activerecord-statementinvalid-pg-notnullviolation-error-null-value-in-column-a_mode-violates-not-null-constraintSean McGivern2017-08-181-1/+13
| |_|_|/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Guess the modes based on the following: 1. If the file didn't exist, it's zero. 2. If the diff contains 'Subproject commit', it might be a submodule, so 0600. 3. Otherwise, it's 0644. This isn't perfect, but it doesn't have to be - it won't change file modes in the repository.
* | | | Move the personal snippet uploads from `system` to `-/system`bvl-rollback-renamed-system-namespaceBob Van Landuyt2017-08-111-0/+79
| | | | | | | | | | | | | | | | | | | | Update the markdown unconditionally since the move might have been done before, but the markdown not updated.
* | | | Merge branch 'split-events-into-push-events' into 'master'Sean McGivern2017-08-111-0/+176
|\ \ \ \ | | | | | | | | | | | | | | | | | | | | Use a separate table for storing push events See merge request !12463
| * | | | Migrate events into a new formatYorick Peterse2017-08-101-0/+176
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit migrates events data in such a way that push events are stored much more efficiently. This is done by creating a shadow table called "events_for_migration", and a table called "push_event_payloads" which is used for storing push data of push events. The background migration in this commit will copy events from the "events" table into the "events_for_migration" table, push events in will also have a row created in "push_event_payloads". This approach allows us to reclaim space in the next release by simply swapping the "events" and "events_for_migration" tables, then dropping the old events (now "events_for_migration") table. The new table structure is also optimised for storage space, and does not include the unused "title" column nor the "data" column (since this data is moved to "push_event_payloads"). == Newly Created Events Newly created events are inserted into both "events" and "events_for_migration", both using the exact same primary key value. The table "push_event_payloads" in turn has a foreign key to the _shadow_ table. This removes the need for recreating and validating the foreign key after swapping the tables. Since the shadow table also has a foreign key to "projects.id" we also don't have to worry about orphaned rows. This approach however does require some additional storage as we're duplicating a portion of the events data for at least 1 release. The exact amount is hard to estimate, but for GitLab.com this is expected to be between 10 and 20 GB at most. The background migration in this commit deliberately does _not_ update the "events" table as doing so would put a lot of pressure on PostgreSQL's auto vacuuming system. == Supporting Both Old And New Events Application code has also been adjusted to support push events using both the old and new data formats. This is done by creating a PushEvent class which extends the regular Event class. Using Rails' Single Table Inheritance system we can ensure the right class is used for the right data, which in this case is based on the value of `events.action`. To support displaying old and new data at the same time the PushEvent class re-defines a few methods of the Event class, falling back to their original implementations for push events in the old format. Once all existing events have been migrated the various push event related methods can be removed from the Event model, and the calls to `super` can be removed from the methods in the PushEvent model. The UI and event atom feed have also been slightly changed to better handle this new setup, fortunately only a few changes were necessary to make this work. == API Changes The API only displays push data of events in the new format. Supporting both formats in the API is a bit more difficult compared to the UI. Since the old push data was not really well documented (apart from one example that used an incorrect "action" nmae) I decided that supporting both was not worth the effort, especially since events will be migrated in a few days _and_ new events are created in the correct format.
* | | | | Fix merge request diff deserialisation when too_large was absentSean McGivern2017-08-111-0/+2
|/ / / / | | | | | | | | | | | | | | | | Not all diffs in st_diffs had the too_large attribute set at all, but the column is non-nullable. Explicitly cast to boolean to avoid that problem.
* | | | Migrate MR commits and diffs to new tablesmerge-request-commits-background-migrationSean McGivern2017-08-031-0/+107
| |_|/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, we stored these as serialised fields - `st_{commits,diffs}` - on the `merge_request_diffs` table. These now have their own tables - `merge_request_diff_{commits,diffs}` - with a column for each attribute of the serialised data. Add a background migration to go through the existing MR diffs and migrate them to the new format. Ignore any contents that cannot be displayed. Assuming that we have 5 million rows to migrate, and each batch of 2,500 rows can be completed in 5 minutes, this will take about 7 days to migrate everything.
* | | Allow groups with the name systembvl-free-system-namespaceBob Van Landuyt2017-07-181-29/+8
| | |
* | | Add a background migration to rename `uploads` in the uploads tableBob Van Landuyt2017-07-181-0/+47
| |/ |/|
* | Simplify stage_id migration as we now use relationsGrzegorz Bizon2017-07-071-9/+4
| |
* | Schedule stage_id bg migrations in batches of 10Grzegorz Bizon2017-07-071-1/+1
| |
* | Schedule stage_id background migration in rangesGrzegorz Bizon2017-07-071-3/+8
| |
* | Sanitize id value passed to async background migrationGrzegorz Bizon2017-07-071-3/+1
| |
* | Improve readability of build stage id migration queryGrzegorz Bizon2017-07-071-6/+7
| |
* | Test if argument passed to a migration is presentGrzegorz Bizon2017-07-071-1/+1
| |
* | Add initial build stage_id ref background migrationGrzegorz Bizon2017-07-071-7/+11
| |
* | Add initial stage_id background migration filesGrzegorz Bizon2017-07-071-0/+16
|/
* Add the ability to perform background migrationsYorick Peterse2017-06-121-0/+0
Background migrations can be used to perform long running data migrations without these blocking a deployment procedure. See MR https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11854 for more information.
View Lorry Controller Status