summaryrefslogtreecommitdiff
path: root/doc/development/uploads
diff options
context:
space:
mode:
authorGitLab Bot <gitlab-bot@gitlab.com>2022-12-20 14:22:11 +0000
committerGitLab Bot <gitlab-bot@gitlab.com>2022-12-20 14:22:11 +0000
commit0c872e02b2c822e3397515ec324051ff540f0cd5 (patch)
treece2fb6ce7030e4dad0f4118d21ab6453e5938cdd /doc/development/uploads
parentf7e05a6853b12f02911494c4b3fe53d9540d74fc (diff)
downloadgitlab-ce-0c872e02b2c822e3397515ec324051ff540f0cd5.tar.gz
Add latest changes from gitlab-org/gitlab@15-7-stable-eev15.7.0-rc42
Diffstat (limited to 'doc/development/uploads')
-rw-r--r--doc/development/uploads/index.md35
-rw-r--r--doc/development/uploads/working_with_uploads.md28
2 files changed, 16 insertions, 47 deletions
diff --git a/doc/development/uploads/index.md b/doc/development/uploads/index.md
index 41ec71451fb..b5509f5934e 100644
--- a/doc/development/uploads/index.md
+++ b/doc/development/uploads/index.md
@@ -40,21 +40,9 @@ When using object storage, administrators can control how those files are moved
This move can happen in one of these ways:
- [Rails controller upload](#rails-controller-upload).
-- [Background upload](#background-upload).
- [Direct upload](#direct-upload).
-These strategies activate as per the following `<feature>.object_store.*` settings:
-
-| | `background_upload` = `false` | `background_upload` = `true` |
-| ------------------------- | ----------------------------- | ------------------------------- |
-| `direct_upload` = `false` | Controller upload | Background upload |
-| `direct_upload` = `true` | Direct upload | Direct upload (takes precedence)|
-
Individual Sidekiq workers might also store files in object storage, which is not something we cover here.
-More importantly, `background_upload` does not imply _all files are uploaded by Sidekiq._
-Sidekiq workers that store files in object storage could still exist when this setting is `false`.
-Those cases are never user-initiated uploads, but they might occur in response to another user-initiated
-action, such as exporting a GitLab repository.
Finally, Workhorse assists most user-initiated uploads using an upload buffering mechanism to keep slow work out of Rails controllers.
This mechanism is explained in [Workhorse assisted uploads](#workhorse-assisted-uploads),
@@ -98,12 +86,12 @@ GitLab to the object store provider. As mentioned above, there are three differe
this HTTP request is sent.
- [Rails controller upload](#rails-controller-upload).
-- [Background upload](#background-upload).
- [Direct upload](#direct-upload).
+- [Workhorse assisted uploads](#workhorse-assisted-uploads).
### Rails controller upload
-When neither background upload nor direct upload are available, Rails uploads the file to object storage
+When direct upload is not available, Rails uploads the file to object storage
as part of the controller `create` action. Which controller is responsible depends on the kind of file uploaded.
A Rails controller upload is very similar to uploading to local storage. The main difference: Rails must
@@ -115,25 +103,6 @@ keep some of the costly I/O work out of Ruby and Rails. Direct upload does a bet
This strategy is only suitable for small file uploads, as it is subject to Puma's 60 second request timeout.
-### Background upload
-
-WARNING:
-This strategy is deprecated in GitLab 14.9 and later, and is scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/26600).
-
-With background uploads enabled:
-
-1. Files are uploaded as if they were to reside in local storage.
-1. When Rails saves the upload metadata and the transaction completes, a Sidekiq job is scheduled.
-1. The Sidekiq job transfers the file to the object store bucket.
- - If the job completes, the upload record is updated to reflect the file's new location.
- - If the job fails or gets lost, the upload stays in local storage and has the lifecycle of a normal local storage upload.
-
-As Rails and Sidekiq must cooperate to move the file to its final destination, it requires shared
-storage and as such is unsuitable for CNG installations. We do not use background upload in GitLab SaaS.
-
-As background upload is an extension of local storage, it benefits from the same [Workhorse assistance](#workhorse-assisted-uploads) to
-keep costly I/O work out of Ruby and Rails.
-
### Direct upload
Direct upload is the recommended way to move large files into object storage in CNG installations like GitLab SaaS.
diff --git a/doc/development/uploads/working_with_uploads.md b/doc/development/uploads/working_with_uploads.md
index c88762e6bd5..a3951fb4c7e 100644
--- a/doc/development/uploads/working_with_uploads.md
+++ b/doc/development/uploads/working_with_uploads.md
@@ -8,9 +8,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w
## Recommendations
-- When creating an uploader, [make it a subclass](#where-should-i-store-my-files) of `AttachmentUploader`
+- When creating an uploader, [make it a subclass](#where-should-you-store-your-files) of `AttachmentUploader`
- Add your uploader to the [tables](#tables) in this document
-- Do not add [new object storage buckets](#where-should-i-store-my-files)
+- Do not add [new object storage buckets](#where-should-you-store-your-files)
- Implement [direct upload](#implementing-direct-upload-support)
- If you need to process your uploads, decide [where to do that](#processing-uploads)
@@ -19,14 +19,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w
- [CarrierWave Uploaders](#carrierwave-uploaders)
- [GitLab modifications to CarrierWave](#gitlab-modifications-to-carrierwave)
-## Where should I store my files?
+## Where should you store your files?
CarrierWave Uploaders determine where files get
stored. When you create a new Uploader class you are deciding where to store the files of your new
feature.
First of all, ask yourself if you need a new Uploader class. It is OK
-to use the same Uploader class for different mountpoints or different
+to use the same Uploader class for different mount points or different
models.
If you do want or need your own Uploader class then you should make it
@@ -160,8 +160,8 @@ we modified it.
The central concept of CarrierWave is the **Uploader** class. The
Uploader defines where files get stored, and optionally contains
validation and processing logic. To use an Uploader you must associate
-it with a text column on an ActiveRecord model. This called "mounting"
-and the column is called the "mountpoint". For example:
+it with a text column on an ActiveRecord model. This is called "mounting"
+and the column is called `mountpoint`. For example:
```ruby
class Project < ApplicationRecord
@@ -169,8 +169,8 @@ class Project < ApplicationRecord
end
```
-Now if I upload an avatar called `tanuki.png` the idea is that in the
-`projects.avatar` column for my project, CarrierWave stores the string
+Now if you upload an avatar called `tanuki.png` the idea is that in the
+`projects.avatar` column for your project, CarrierWave stores the string
`tanuki.png`, and that the AttachmentUploader class contains the
configuration data and directory schema. For example if the project ID
is 123, the actual file may be in
@@ -179,12 +179,12 @@ The directory
`/var/opt/gitlab/gitlab-rails/uploads/-/system/project/avatar/123/`
was chosen by the Uploader using among others configuration
(`/var/opt/gitlab/gitlab-rails/uploads`), the model name (`project`),
-the model ID (`123`) and the mountpoint (`avatar`).
+the model ID (`123`) and the mount point (`avatar`).
> The Uploader determines the individual storage directory of your
-> upload. The mountpoint column in your model contains the filename.
+> upload. The `mountpoint` column in your model contains the filename.
-You never access the mountpoint column directly because CarrierWave
+You never access the `mountpoint` column directly because CarrierWave
defines a getter and setter on your model that operates on file handle
objects.
@@ -213,7 +213,7 @@ CarrierWave has 2 storage engines:
|CarrierWave class|GitLab name|Description|
|---|---|---|
-|`CarrierWave::Storage::File`|`ObjectStorage::Store::LOCAL` |Local files, accessed through the Ruby stdlib|
+|`CarrierWave::Storage::File`|`ObjectStorage::Store::LOCAL` |Local files, accessed through the Ruby `stdlib` |
| `CarrierWave::Storage::Fog`|`ObjectStorage::Store::REMOTE`|Cloud files, accessed through the [Fog gem](https://github.com/fog/fog)|
GitLab uses both of these engines, depending on configuration.
@@ -227,8 +227,8 @@ storage engine file by file.
An Uploader is associated with two storage areas: regular storage and
cache storage. Each has its own storage engine. If you assign a file
-to a mountpoint setter (`project.avatar =
-File.open('/tmp/tanuki.png')`) you will copy/move the file to cache
+to a mount point setter (`project.avatar = File.open('/tmp/tanuki.png')`)
+you will copy/move the file to cache
storage as a side effect via the `cache!` method. To persist the file
you must somehow call the `store!` method. This either happens via
[ActiveRecord callbacks](https://github.com/carrierwaveuploader/carrierwave/blob/v1.3.2/lib/carrierwave/orm/activerecord.rb#L55)