summaryrefslogtreecommitdiff
path: root/spec/uploaders
Commit message (Collapse)AuthorAgeFilesLines
* Queries for Upload should be scoped by modelAdam Hegyi2019-07-111-0/+21
|
* Use #filename when generating upload URLs53357-fix-plus-in-upload-file-namesHeinrich Lee Yu2019-07-041-26/+36
| | | | We don't need to find the filename from the remote URL
* Fix order-dependent spec failure in appearance_spec.rbStan Hu2019-07-031-2/+1
| | | | | | | | | When file_mover_spec.rb ran, it would initialize fog-aws with `Fog::AWS::Storage::Real` service instead of `Fog::AWS::Storage::Mock` because `Fog.mock!` was not called. Ensure that we use `stub_uploads_object_storage` to prevent that from happening. Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/64083
* Support object storage at FileMover classOswaldo Ferreira2019-06-301-36/+83
|
* Persist tmp snippet uploadsOswaldo Ferreira2019-06-171-12/+26
| | | | | | | | | | It persist temporary personal snippets under user/:id namespaces temporarily while creating a upload record to track it. If an user gets removed while it's still a tmp upload, it also gets removed. If the tmp upload is sent, the upload gets moved to personal_snippets/:id as before. The upload record also gets updated to the new model type as well.
* Remove legacy artifact related coderemove-legacy-artifacts-related-codeShinya Maeda2019-05-312-95/+0
| | | | | | We've already migrated all the legacy artifacts to the new realm, which is ci_job_artifacts table. It's time to remove the old code base that is no longer used.
* Fix incorrect prefix used in new uploads for personal snippetsStan Hu2019-05-161-20/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24550 fixed the case where the wrong path on disk was being searched, but it inadvertently ommitted the `/uploads/-/system` prefix when rendering the Markdown for personal snippet uploads when they were stored directly in object storage. A personal snippet path is stored using FileUploader#upload_path. The format for the path: Local storage: :random_hex/:filename. Object storage: personal_snippet/:id/:random_hex/:filename. upload_paths represent the possible paths for a given identifier, which will vary depending on whether the file is stored in local or object storage. upload_path should match an element in upload_paths. base_dir represents the path seen by the user in Markdown, and it should always be prefixed with uploads/-/system. store_dirs represent the paths that are actually used on disk. For object storage, this should omit the prefix /uploads/-/system. For example, consider the requested path /uploads/-/system/personal_snippet/172/ff4ad5c2/file.png. For local storage: base_dir: uploads/-/system/personal_snippet/172 upload_path: ff4ad5c2/file.png upload_paths: ["ff4ad5c2/file.png", "personal_snippet/172/ff4ad5c2/file.png"]. store_dirs: {1=>"uploads/-/system/personal_snippet/172/ff4ad5c2", 2=>"personal_snippet/172/ff4ad5c2"} For object storage: upload_path: personal_snippet/172/ff4ad5c2/file.png Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/61671
* Optimise upload path callsSean McGivern2019-05-141-1/+1
| | | | | | | | String#underscore isn't particularly slow, but it's possible for us to call it many times in a users autocomplete request, with mostly-static values ('User', 'Group', etc.). We can memoise this and save a surprising amount of time (around 10% of the total request time in some cases).
* Clean up CarrierWave's import/export filesStan Hu2019-05-071-0/+32
| | | | | | | | | | | | | | | | | | | | | | | Unlike uploads that have been uploaded with Tempfile, the project import/export archives are stored in a temporary cache directory and remain there if: 1. Object storage is enabled 2. `move_to_store` is set to `true`. CarrierWave will leave these files there until disk space runs out or a clean step is run manually. If `move_to_store` is set to `false`, CarrierWave will remove the files after storing them. However, unlike a local file, with object storage, the file is still copied, so setting `move_to_store` to `true` doesn't buy us anything. To ensure files are cleaned up, we can just inherit from the GitlabUploader implementation of `move_to_store`, which returns `true` if it's a local file, `false` otherwise. Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/60656
* Disable method replacement in avatar loadingStan Hu2019-04-291-0/+8
| | | | | | | | | | | | | | | | We've seen a significant performance penalty when using `BatchLoader#__replace_with!`. This defines methods on the batch loader that proxy to the 'real' object using send. The alternative is `method_missing`, which is slower. However, we've noticed that `method_missing` can be faster if: 1. The objects being loaded have a large interface. 2. We don't call too many methods on the loaded object. Avatar uploads meet both criteria above, so let's use the newly-released feature in https://github.com/exAspArk/batch-loader/pull/45. Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/60903
* Autocorrect with RSpec/ExampleWording copThong Kuah2019-04-051-2/+2
| | | | | | | - rewords examples starting with 'should' - rewords examples starting with 'it' Note: I had to manually fixup "onlies" to "only"
* Speed up avatar URLs with object storageSean McGivern2019-04-041-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | With object storage enabled, calling `#filename` on an upload does this: 1. Call the `#filename` method on the CarrierWave object. 2. Generate the URL for that object. 3. If the uploader isn't public, do so by generating an authenticated URL, including signing that request. That's all correct behaviour, but for the case where we use `#filename`, it's typically to generate a GitLab URL. That URL doesn't need to be signed because we do our own auth. Signing the URLs can be very expensive, especially in batch (say, we need to get the avatar URLs for 150 users in one request). It's all unnecessary work. If we used the `RecordsUploads` concern, we have already recorded a `path` in the database. That `path` is actually generated from CarrierWave's `#filename` at upload time, so we don't need to recompute it - we can just use it and strip off the prefix if it's available. On a sample users autocomplete URL, at least 10% of the time before this change went to signing URLs. After this change, we spend no time in URL signing, and still get the correct results.
* Don't try to set any ACL on uploaded objectsBastian Blank2019-03-301-1/+1
| | | | | | | | Set ACL of uploaded objects to not-public (fog-aws) or no ACL at all (fog-google). Value is ignored by other supported backends (fog-aliyun, fog-openstack, fog-rackspace) This allows uploads to GCS with ACL support disabled.
* Check snippet attached file to be moved is within designated directoryMark Chao2019-02-211-0/+30
| | | | Previously one could move any temp/ sub folder around.
* Align spec with actual usageMark Chao2019-02-131-2/+1
| | | | Currently we pass temp file path to FileMover
* Allow MR diffs to be placed into an object storeNick Thomas2019-02-051-0/+67
|
* Enable the Layout/ExtraSpacing cop56392-enable-the-layout-extraspacing-copRémy Coutable2019-01-242-2/+2
| | | | Signed-off-by: Rémy Coutable <remy@rymai.me>
* Fix 404s with snippet uploads in object storageStan Hu2019-01-211-17/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | Previously, an HTTP request for `/uploads/-/system/personal_snippet/:snippet_id/:hash/:filename` would look for an uploader of `PersonalFileUploader` class and use `PersonalFileUploader#upload_paths` to search the datbase for one of the following paths: 1. `:hash/:filename` 2. `uploads/-/system/personal_snippet/:id/:hash/:filename` However, when the upload were stored in object storage, `PersonalFileUploader#store_dirs` stored the path as: `personal_snippet/:snippet_id/:hash` The extraneous `uploads/-/system` prefix prevented the path from being matched, and uploads in object storage would return a 404 error. Uploads in local storage would work fine. To fix this, we set the `#base_dir` properly so that `#upload_paths` generates the right value for object storage. Note that this also makes `#store_dirs` do the right thing in `FileUploader`. Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/52595
* Avoid extra storage bucket perm and querymk/avoid-extra-storage-perm-and-queryMichael Kozono2018-12-212-2/+2
| | | | | | | | | | | Specifically, the `ListAllMyBuckets` permission. This works if you know the directory exists. See more: * https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23981 * https://stackoverflow.com/a/12288581/1992201
* Fix already initialized constant IDENTIFIER warningSemyon Pupkov2018-12-102-18/+36
| | | | | /app/spec/uploaders/personal_file_uploader_spec.rb:3: warning: already initialized constant IDENTIFIER /app/spec/uploaders/namespace_file_uploader_spec.rb:3: warning: previous definition of IDENTIFIER was here
* Extract code from Issues::MoveServicece-3777-promote-to-epicJarka Košanová2018-11-122-5/+80
| | | | | | | Extract code to make it easier reusable - introduce AttributesRewriter and ContentRewriter - support group entites when rewriting content - make Uploader copy_to working for Namespaces
* Merge branch 'fix-namespace-upload' into 'master'Kamil Trzciński2018-09-111-0/+6
|\ | | | | | | | | | | | | Fix workhorse temp path for namespace uploads Closes gitlab-ee#7009 See merge request gitlab-org/gitlab-ce!21650
| * Fix workhorse temp path for namespace uploadsJarka Košanová2018-09-111-0/+6
| |
* | fix avatar uploader errorJames Lopez2018-09-101-0/+8
|/
* Merge branch 'fix-namespace-uploader' into 'master'Kamil Trzciński2018-09-071-0/+14
|\ | | | | | | | | Fix NamespaceUploader.base_dir for remote uploads See merge request gitlab-org/gitlab-ce!21308
| * Fix NamespaceUploader.base_dir for remote uploadsfix-namespace-uploaderJarka Kadlecová2018-08-211-0/+14
| |
* | Merge branch 'add-background-migration-for-legacy-traces' into 'master'Kamil Trzciński2018-09-041-0/+47
|\ \ | | | | | | | | | | | | | | | | | | Migrate job artifacts data from `ci_builds` to `ci_job_artifacts` table (with Background migrations) Closes #46652 See merge request gitlab-org/gitlab-ce!18615
| * | Squashed commit of the following:add-background-migration-for-legacy-tracesShinya Maeda2018-08-271-0/+47
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | commit 8ea114e5c349ad23f7293cf8141798aa6c9384e3 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Aug 17 13:32:14 2018 +0900 Remove redandant parentheses commit 3eb34f9305980701f0c1528f95fd615c1ca4d50d Merge: 24308e4d1da 46494f46a13 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Aug 17 13:22:05 2018 +0900 Merge branch 'master-ce' into add-background-migration-for-legacy-traces commit 24308e4d1dab6fbf8437afc56fff7b83cc7ce534 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Aug 16 16:39:20 2018 +0900 Fix spec commit 5a63312d1b36296440da6a874bb667b7dc06869c Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Aug 16 14:46:59 2018 +0900 Format SQL in MIgrateLegacyArtifacts class commit 8894196d2aa2f9bf153cb03fef9603858478db70 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Aug 16 14:31:18 2018 +0900 Remove unnecessary index commit d9753ac058fc48c8f594ee90064f675f7a869b17 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Aug 16 14:24:20 2018 +0900 Decouple pertial index to a different migration file commit 1ce4a9bec6a4f0f981f4139c1a146ca7d887e6ba Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Aug 16 14:14:03 2018 +0900 Simplified adding file_location column commit 47b101fae914470dc4d048f6d33c509ec8270656 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Aug 16 14:11:42 2018 +0900 Remove add_column_with_default commit 80ec4f774f50bbf66e65f283f8fba541c6334e1c Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Aug 16 14:02:46 2018 +0900 Simplified job_artifact_uploader commit d835dab36e85f4fc7a7d0735ede0f0ea081178bf Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Aug 16 13:59:03 2018 +0900 Elaborate comments on `file_location` commit 6e78f23683bbae1372783f19179206369d7fbae7 Merge: fcef07ef361 79fdfec51fa Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Aug 16 13:39:28 2018 +0900 Merge branch 'master-ce' into add-background-migration-for-legacy-traces commit fcef07ef361080823432071722897321b2dd8354 Merge: 1eac3083194 bf9fd9c3fc3 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Jul 6 21:46:22 2018 +0900 Merge branch 'master' into add-background-migration-for-legacy-traces commit 1eac30831947cceec7610be6acc7d3bffadaf617 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Jul 6 21:45:49 2018 +0900 Use SQL instead of strip_heredoc commit 08b17e517722e1e4ef8b60e753d53d697953a293 Merge: c0f48ff243d dc478a82432 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Jul 5 23:20:37 2018 +0900 Merge branch 'master' into add-background-migration-for-legacy-traces commit c0f48ff243ddbe8af6c63aa0170538717191f44a Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Jul 5 17:37:40 2018 +0900 Fix flaky spec commit d082dab7cb456f9b31e76f49d1db46b606ba13ce Merge: 8d6059d08ff 116955c4533 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Jul 5 17:35:06 2018 +0900 Merge branch 'master' into add-background-migration-for-legacy-traces commit 8d6059d08fffaa99689afe32b66d93b538fb62f2 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Jul 4 21:39:09 2018 +0900 Fix static analysys failure commit c4bdb18d582c520723a73c4c90c522332fa94d36 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Jul 4 20:16:16 2018 +0900 Cleanup spec commit 469e826429feb6a8ce66eb5d5e087f7182eb8813 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Jul 4 19:09:12 2018 +0900 Clean up migrate_legacy_artifacts spec commit dee84be650ad5afc9f18ac05eccb0736e4b1c121 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Jul 4 18:13:19 2018 +0900 Set default value at the database level commit b32cb2f8a7e03238cf97d90f9e05ca11b5c8b687 Merge: 83f7c970ad8 4c1a2a9b997 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Jul 4 17:06:02 2018 +0900 Merge branch 'master' into add-background-migration-for-legacy-traces commit 83f7c970ad821f1b52999398070facf59f320479 Merge: 823629c1d9a 275fbf24b18 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Jul 3 18:39:36 2018 +0900 Merge branch 'master' into add-background-migration-for-legacy-traces commit 823629c1d9aac5e7dcfb6bc393557a72ca3370b4 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Jul 2 14:59:54 2018 +0900 Decouple background migrations of filling nullified file_locations commit 42ec60e8a74478c62962f5c66abfbe3e908d9e49 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Jul 2 14:46:38 2018 +0900 Unify partial index migrations commit c145e9306419f704b14e2a92f51a9cf2d105f7e7 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Jul 2 14:24:14 2018 +0900 Make file_location non-null values commit a442456f6a83327b2736f49b9522084b5675d129 Merge: c2cafa1051c 3cf683629e6 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Jul 2 13:16:29 2018 +0900 Merge branch 'master' into add-background-migration-for-legacy-traces commit c2cafa1051cb7a4582c93b653e0ef3506ee11af4 Merge: 8e5f4f88410 e38db196598 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Jun 28 17:13:48 2018 +0900 Merge branch 'master' into add-background-migration-for-legacy-traces commit 8e5f4f884107c0f574545f043fbacea2698ef3a4 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Jun 27 18:54:03 2018 +0900 Remove unneccesary specs commit 86c1c68b1000770fcf8086fa71801b332fb6df5a Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Jun 27 18:05:36 2018 +0900 Add a partial index on ci_builds.id for legacy artifacts search commit 70aa08c89bb6cd9b12d38b64cbf1838fb919c30f Merge: 37801122986 292cf668905 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Jun 27 17:15:33 2018 +0900 Merge branch 'master' into add-background-migration-for-legacy-traces commit 378011229864c1e056cf995444f947f6b352172c Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Jun 7 14:18:00 2018 +0900 Clean up migration code. Defining migration custom class in only post migration file which requires it for each_batch commit 5cfe73318b91f377897e9fc50d67145093846459 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Jun 6 23:57:03 2018 +0900 Remove indexing for mysql. with_legacy_artifacts targets all archive rows. Enhance tests. commit 696c030d4a1ae2e1ee5e0eac771e574191fcb477 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Jun 6 19:59:35 2018 +0900 Fix static analysys. Split methods. Use `file_types` commit 86217c650c99d86d5b329229799f250f2ac5fbf6 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Jun 6 18:55:15 2018 +0900 Use raw sql for better readability commit df449404cb167edd61af1e59ca24f25db0a9ca9f Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Jun 6 17:47:39 2018 +0900 Revert unneccesary change commit 764977a1d98b34e34fcd229fca72c9595d62861a Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Jun 6 16:09:14 2018 +0900 Remove AddIndexToCiBuildsArtifactsFile. Add temporary index in background migration class. commit 7728ab3f9dba0af21033e1ca3b0213cacaef85d8 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Jun 6 15:28:50 2018 +0900 Try to explicitly separate with postgresql and mysql for addin index commit b95b47b965413f6590bd7eb38e34b4ef88cd2002 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Jun 6 14:29:18 2018 +0900 Dry up code by referring `BackgroundMigration` namespace. Use `BETWEEN` raw SQL to make the edge case explicit. commit 5b404e46e2a7c0d5a49b9745a8d985a80c9e380a Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Jun 6 14:15:37 2018 +0900 Use `store_path` to make spec more explicit commit 9f80de8e5aed49aae148278b0d27273e92ddbf25 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Jun 6 14:03:26 2018 +0900 Use length not limit commit f78503f3e55cc08d78274767b9381631201262e0 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Jun 6 13:48:51 2018 +0900 Add length back for indexing commit b96a57d5e5c7d2a97a1926ec95d67f0a2a1b8c93 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Jun 5 12:42:14 2018 +0000 Update schema.rb commit 9c25acc9752e5cf87f4062dc9f33fb609e8fb7be Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Jun 5 12:40:48 2018 +0000 Add the guard clause in #down method too commit 2bd5dadbd1b742d15bfb0914ad1be7e2182c1755 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Jun 5 19:41:08 2018 +0900 Add a gurad caluse to block adding concurrent index for mysql commit 9a5e12768ed30f75dd796d7c70f54b5e78aaa746 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Jun 5 19:20:33 2018 +0900 Specify length of index in schema.rb manually for resolving mysql errors commit c0ec68af02706d3c01b20b9f589113a900ff2071 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Jun 5 17:51:40 2018 +0900 Separate add_concurrent_index by database type commit b5f4ed37d9f49719fd8235a7069bb93f3c73c2dc Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Jun 5 17:30:19 2018 +0900 Fix length size to use TEXT commit b38b56056f4de384e32ee9acfbf161397de079fd Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Jun 5 17:20:35 2018 +0900 Fix schema version for background migration spec commit 32c48ef7cba61ceb285c6c7c1f48964a204dff20 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Jun 5 16:22:00 2018 +0900 Specify length of indexed column. (This is neccessary for TEXT type) commit a2e975f2e610ffd5f7f400b29a77d6b337005f1d Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Jun 5 15:13:42 2018 +0900 Fix spec; migrate_legacy_artifacts_to_job_artifacts_spec.rb commit c78c1e3f127b947e46a007bec2d56e0f3a55d7fa Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Jun 4 20:15:23 2018 +0900 Separate add_concurrent_index migration from background migration commit ad55e33bc2cd622b19f78ace64a69c18870d62df Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Jun 4 19:59:14 2018 +0900 Revert "Remove index_exists? guard" This reverts commit d76807838cf06e023b608c16426174a752a8ccf5. commit b2693be76fb2d643058f7f0fffd0d75ef13c38f0 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Jun 4 17:11:47 2018 +0900 Remove index_exists? guard commit a2eb053d3b3bd8c8ad49958d58b7225658214a72 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Jun 4 15:41:54 2018 +0900 Wrap insertion and deletion in a single transaction commit d4bb2c709ce549eb5d7382c03390bfdf6ec90297 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Jun 4 15:08:17 2018 +0900 Add temporary index to ci_builds.artifacts_file to speed up SELECT query commit 33d62be9ee72a5ee4a61a7bc793df84ea7e85730 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon May 28 17:31:38 2018 +0900 Add test for legacy path proxy commit 57ab71a05b464dc4f477837f237eb15e8e51bb84 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon May 28 15:59:36 2018 +0900 Optimize queries. Add some tests for filtering logic. commit 8f24dfaac70bc564973bd81745b80421f1a3e1af Author: Shinya Maeda <shinya@gitlab.com> Date: Mon May 28 15:30:38 2018 +0900 Fix static analysis commit ccc60ddd6076f997f63f46cad814e4ac9e4cbdfc Author: Shinya Maeda <shinya@gitlab.com> Date: Thu May 24 16:51:59 2018 +0900 Add changelog commit fdbdb07a40a8343e8292e618f90658e77cc2115e Author: Shinya Maeda <shinya@gitlab.com> Date: Thu May 24 16:50:55 2018 +0900 Add spec for migrations commit 04886af8724258f9971798adcf3eacd5716f20df Author: Shinya Maeda <shinya@gitlab.com> Date: Thu May 24 14:48:31 2018 +0900 Revise comments commit 55bcbee0576b5ac08b4ed461b85d059e0c670159 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu May 24 14:40:29 2018 +0900 Revise comments commit db6628675abfe24cc10e682a65bf87ca02504deb Author: Shinya Maeda <shinya@gitlab.com> Date: Tue May 15 17:27:12 2018 +0900 Add spec base (Not finished) commit e92dcc392eb4ad5aac06a3510a5879a3f084b9ee Author: Shinya Maeda <shinya@gitlab.com> Date: Tue May 15 17:26:49 2018 +0900 Added artifacts_metadata_store to remove in background migration commit 91388399d80b58a831b21e6582d5f5ab29e785c7 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue May 15 17:26:20 2018 +0900 Add a condition to exclude jobs which have job_artifacts already commit e084ce1181d07d0799714eee058d6c9182bc2fcf Author: Shinya Maeda <shinya@gitlab.com> Date: Tue May 15 16:51:15 2018 +0900 Clean up migrate_legacy_artifacts.rb (Fix static analysys) commit ca4e5d33e27690dc276ba87d08dc24bcf31267f4 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue May 15 16:32:59 2018 +0900 Remove the culcuration method of metadata size commit eac4c75be932ae6936702ddfd9202f333f18053f Author: Shinya Maeda <shinya@gitlab.com> Date: Tue May 15 16:11:16 2018 +0900 Remove Gitlab::BackgroundMigration:: scope commit 4a5ca96b628a59eeba25778569e719695ac8b5e8 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue May 15 16:09:09 2018 +0900 Add limit to file_location column commit 49ff3d9c98dc145fe3c91cd8ba39d7d71d183fb7 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue May 15 16:05:22 2018 +0900 Use size.to_i instead of if-condition commit 526656e6ee5683ec7ea07bd6af3438c3fc32375d Author: Shinya Maeda <shinya@gitlab.com> Date: Tue May 15 16:02:57 2018 +0900 Rename current_path to hashed_path commit 8e6faca7900cf382a35323b2ed0fabede9bf3bf9 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue May 8 16:53:43 2018 +0900 Rename location_1/2 to current/legacy_path commit 5034543255963a250e076f37c5e42fbf4cb0fd05 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue May 1 21:44:10 2018 +0900 Rename path_type to file_location. Fix JobArtifactUploader to look up correct file location. commit 4f7e3503f13d02d7a70e7b68bf740a8d8bb525db Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Apr 27 17:33:15 2018 +0900 Add background migration for legacy artifacts Add default_value_for to file_location column Set file_location explicityly with default_values_for Add file_location to factory Remove tests of default_values_for Set file_location nil as default
* | Uploads to wiki stored inside the wiki git repositoryFrancisco Javier López2018-09-041-21/+4
|/
* Fix filename for accelerated uploadsJan Provaznik2018-07-181-0/+46
|
* Merge branch ↵Sean McGivern2018-07-161-0/+9
|\ | | | | | | | | | | | | | | | | '48745-project-exports-fail-when-uploads-have-been-migrated-to-object-storage' into 'master' Resolve "Project exports fail when uploads have been migrated to object storage" Closes #48745 See merge request gitlab-org/gitlab-ce!20484
| * use fileuploader dynamic path method in uploads manager and add spec48745-project-exports-fail-when-uploads-have-been-migrated-to-object-storageJames Lopez2018-07-131-0/+9
| |
* | Fix and add specs for testing metadata entryKamil Trzciński2018-07-092-37/+62
| |
* | Access metadata directly from Object StorageKamil Trzciński2018-07-091-1/+1
|/ | | | Previously we would pull the file, now, we just stream-it as needed from Object Storage
* Update Import/Export to use object storage (based on aa feature flag)James Lopez2018-07-061-0/+20
|
* Fix minor spec review comments in uploader specsStan Hu2018-07-031-3/+3
|
* add support for file copy on object storageMicaël Bergeron2018-07-031-0/+44
|
* Bring this test from EEunify-object_storage_spec.rbLin Jen-Shin2018-06-211-0/+12
|
* Remove FaviconUploader favicon_main versiondm-remove-favicon-main-versionDouwe Maan2018-06-151-29/+0
| | | | See https://gitlab.com/gitlab-org/gitlab-ce/issues/47677 for more information
* Merge branch ↵Sean McGivern2018-06-131-0/+12
|\ | | | | | | | | | | | | | | | | '47513-upload-migration-lease-key-is-incorrect-for-non-mounted-uploaders' into 'master' Resolve "Upload migration lease key is incorrect for non-mounted uploaders" Closes #47513 See merge request gitlab-org/gitlab-ce!19600
| * Use upload ID instead of model ID in lease keyJan Provaznik2018-06-121-0/+12
| | | | | | | | | | | | | | For FileUploaders it's possible that a model has many uploads and if lease key is created only from model id, it causes that the model's uploads can not be migrated in parallel because the exclusive lease key would be same for all uploads of the model.
* | Merge branch '47408-migrateuploadsworker-is-doing-n-1-queries-on-migration' ↵Sean McGivern2018-06-132-15/+40
|\ \ | |/ |/| | | | | | | | | | | | | into 'master' Resolve "`MigrateUploadsWorker` is doing N+1 queries on migration" Closes #47408 See merge request gitlab-org/gitlab-ce!19547
| * apply feedback47408-migrateuploadsworker-is-doing-n-1-queries-on-migrationMicaël Bergeron2018-06-121-20/+14
| |
| * apply feedbackMicaël Bergeron2018-06-081-2/+2
| |
| * fix the failing specMicaël Bergeron2018-06-071-2/+2
| |
| * shave off another 20% queryMicaël Bergeron2018-06-071-2/+1
| |
| * shave off 30% of the query countMicaël Bergeron2018-06-061-9/+41
| |
* | Don't use Rails.root.join for upload fixture pathsRémy Coutable2018-06-111-1/+1
| | | | | | | | Signed-off-by: Rémy Coutable <remy@rymai.me>
* | Merge branch 'rs-update-rspec' into 'master'Rémy Coutable2018-06-1111-16/+13
|\ \ | | | | | | | | | | | | Update rspec to 3.7 See merge request gitlab-org/gitlab-ce!19449
| * | Correct more usages of non-relative `fixture_file_upload` pathsRobert Speicher2018-06-073-8/+5
| | |