summaryrefslogtreecommitdiff
path: root/doc/development/geo/framework.md
diff options
context:
space:
mode:
Diffstat (limited to 'doc/development/geo/framework.md')
-rw-r--r--doc/development/geo/framework.md147
1 files changed, 135 insertions, 12 deletions
diff --git a/doc/development/geo/framework.md b/doc/development/geo/framework.md
index de4d6fb869d..64c9030e3dd 100644
--- a/doc/development/geo/framework.md
+++ b/doc/development/geo/framework.md
@@ -90,6 +90,14 @@ module Geo
def self.model
::Packages::PackageFile
end
+
+ # Change this to `true` to release replication of this model. Then remove
+ # this override in the next release.
+ # The feature flag follows the format `geo_#{replicable_name}_replication`,
+ # so here it would be `geo_package_file_replication`
+ def self.replication_enabled_by_default?
+ false
+ end
end
end
```
@@ -180,6 +188,11 @@ For example, to add support for files referenced by a `Widget` model with a
mount_uploader :file, WidgetUploader
+ def local?
+ # Must to be implemented, Check the uploader's storage types
+ file_store == ObjectStorage::Store::LOCAL
+ end
+
def self.replicables_for_geo_node
# Should be implemented. The idea of the method is to restrict
# the set of synced items depending on synchronization settings
@@ -206,10 +219,30 @@ For example, to add support for files referenced by a `Widget` model with a
def carrierwave_uploader
model_record.file
end
+
+ # Change this to `true` to release replication of this model. Then remove
+ # this override in the next release.
+ # The feature flag follows the format `geo_#{replicable_name}_replication`,
+ # so here it would be `geo_widget_replication`
+ def self.replication_enabled_by_default?
+ false
+ end
end
end
```
+1. Add this replicator class to the method `replicator_classes` in
+`ee/lib/gitlab/geo.rb`:
+
+ ```ruby
+ def self.replicator_classes
+ classes = [::Geo::PackageFileReplicator,
+ ::Geo::WidgetReplicator]
+
+ classes.select(&:enabled?)
+ end
+ ```
+
1. Create `ee/spec/replicators/geo/widget_replicator_spec.rb` and perform
the setup necessary to define the `model_record` variable for the shared
examples.
@@ -226,13 +259,15 @@ For example, to add support for files referenced by a `Widget` model with a
end
```
-1. Create the `widget_registry` table so Geo secondaries can track the sync and
- verification state of each Widget's file:
+1. Create the `widget_registry` table, with columns ordered according to [our guidelines](../ordering_table_columns.md) so Geo secondaries can track the sync and
+ verification state of each Widget's file. This migration belongs in `ee/db/geo/migrate`:
```ruby
# frozen_string_literal: true
class CreateWidgetRegistry < ActiveRecord::Migration[6.0]
+ include Gitlab::Database::MigrationHelpers
+
DOWNTIME = false
disable_ddl_transaction!
@@ -244,12 +279,12 @@ For example, to add support for files referenced by a `Widget` model with a
t.integer :widget_id, null: false
t.integer :state, default: 0, null: false, limit: 2
t.integer :retry_count, default: 0, limit: 2
- t.text :last_sync_failure
t.datetime_with_timezone :retry_at
t.datetime_with_timezone :last_synced_at
t.datetime_with_timezone :created_at, null: false
+ t.text :last_sync_failure
- t.index :widget_id
+ t.index :widget_id, name: :index_widget_registry_on_widget_id
t.index :retry_at
t.index :state
end
@@ -286,6 +321,8 @@ For example, to add support for files referenced by a `Widget` model with a
1. Update `REGISTRY_CLASSES` in `ee/app/workers/geo/secondary/registry_consistency_worker.rb`.
+1. Add `widget_registry` to `ActiveSupport::Inflector.inflections` in `config/initializers_before_autoloader/000_inflections.rb`.
+
1. Create `ee/spec/factories/geo/widget_registry.rb`:
```ruby
@@ -343,8 +380,13 @@ Widgets should now be replicated by Geo!
#### Verification
-1. Add verification state fields to the `widgets` table so the Geo primary can
- track verification state:
+There are two ways to add verification related fields so that the Geo primary
+can track verification state:
+
+##### Option 1: Add verification state fields to the existing `widgets` table itself
+
+1. Add a migration to add columns ordered according to [our guidelines](../ordering_table_columns.md)
+for verification state to the widgets table:
```ruby
# frozen_string_literal: true
@@ -353,17 +395,39 @@ Widgets should now be replicated by Geo!
DOWNTIME = false
def change
- add_column :widgets, :verification_retry_at, :datetime_with_timezone
- add_column :widgets, :verified_at, :datetime_with_timezone
- add_column :widgets, :verification_checksum, :binary, using: 'verification_checksum::bytea'
- add_column :widgets, :verification_failure, :string
- add_column :widgets, :verification_retry_count, :integer
+ change_table(:widgets) do |t|
+ t.integer :verification_retry_count, limit: 2
+ t.column :verification_retry_at, :datetime_with_timezone
+ t.column :verified_at, :datetime_with_timezone
+ t.binary :verification_checksum, using: 'verification_checksum::bytea'
+
+ # rubocop:disable Migration/AddLimitToTextColumns
+ t.text :verification_failure
+ # rubocop:enable Migration/AddLimitToTextColumns
+ end
+ end
+ end
+ ```
+
+1. Adding a `text` column also [requires](../database/strings_and_the_text_data_type.md#add-a-text-column-to-an-existing-table)
+ setting a limit:
+
+ ```ruby
+ # frozen_string_literal: true
+
+ class AddVerificationFailureLimitToWidgets < ActiveRecord::Migration[6.0]
+ DOWNTIME = false
+
+ disable_ddl_transaction!
+
+ def change
+ add_text_limit :widgets, :verification_failure, 255
end
end
```
1. Add a partial index on `verification_failure` and `verification_checksum` to ensure
- re-verification can be performed efficiently:
+ re-verification can be performed efficiently. Add a migration in `ee/db/geo/migrate/`:
```ruby
# frozen_string_literal: true
@@ -387,6 +451,65 @@ Widgets should now be replicated by Geo!
end
```
+##### Option 2: Create a separate `widget_states` table with verification state fields
+
+1. Add a migration in `ee/db/geo/migrate/` to create a `widget_states` table and add a
+ partial index on `verification_failure` and `verification_checksum` to ensure
+ re-verification can be performed efficiently. Order the columns according to [our guidelines](../ordering_table_columns.md):
+
+ ```ruby
+ # frozen_string_literal: true
+
+ class CreateWidgetStates < ActiveRecord::Migration[6.0]
+ include Gitlab::Database::MigrationHelpers
+
+ DOWNTIME = false
+
+ disable_ddl_transaction!
+
+ def up
+ unless table_exists?(:widget_states)
+ with_lock_retries do
+ create_table :widget_states, id: false do |t|
+ t.references :widget, primary_key: true, null: false, foreign_key: { on_delete: :cascade }
+ t.datetime_with_timezone :verification_retry_at
+ t.datetime_with_timezone :verified_at
+ t.integer :verification_retry_count, limit: 2
+ t.binary :verification_checksum, using: 'verification_checksum::bytea'
+ t.text :verification_failure
+
+ t.index :verification_failure, where: "(verification_failure IS NOT NULL)", name: "widgets_verification_failure_partial"
+ t.index :verification_checksum, where: "(verification_checksum IS NOT NULL)", name: "widgets_verification_checksum_partial"
+ end
+ end
+ end
+
+ add_text_limit :widget_states, :verification_failure, 255
+ end
+
+ def down
+ drop_table :widget_states
+ end
+ end
+ ```
+
+1. Add the following lines to the `widget` model:
+
+ ```ruby
+ class Widget < ApplicationRecord
+ ...
+ has_one :widget_state, inverse_of: :widget
+
+ delegate :verification_retry_at, :verification_retry_at=,
+ :verified_at, :verified_at=,
+ :verification_checksum, :verification_checksum=,
+ :verification_failure, :verification_failure=,
+ :verification_retry_count, :verification_retry_count=,
+ to: :widget_state
+ ...
+ end
+ ```
+
To do: Add verification on secondaries. This should be done as part of
[Geo: Self Service Framework - First Implementation for Package File verification](https://gitlab.com/groups/gitlab-org/-/epics/1817)