diff options
Diffstat (limited to 'doc/development/geo/framework.md')
| -rw-r--r-- | doc/development/geo/framework.md | 68 |
1 files changed, 52 insertions, 16 deletions
diff --git a/doc/development/geo/framework.md b/doc/development/geo/framework.md index e440e324c4a..e4518ce1b57 100644 --- a/doc/development/geo/framework.md +++ b/doc/development/geo/framework.md @@ -1,12 +1,12 @@ --- stage: Enablement group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Geo self-service framework -NOTE: **Note:** +NOTE: This document is subject to change as we continue to implement and iterate on the framework. Follow the progress in the [epic](https://gitlab.com/groups/gitlab-org/-/epics/2161). If you need to replicate a new data type, reach out to the Geo @@ -393,6 +393,8 @@ can track verification state. def change change_table(:widgets) do |t| + t.integer :verification_state, default: 0, limit: 2, null: false + t.column :verification_started_at, :datetime_with_timezone t.integer :verification_retry_count, limit: 2 t.column :verification_retry_at, :datetime_with_timezone t.column :verified_at, :datetime_with_timezone @@ -431,36 +433,53 @@ can track verification state. end ``` -1. Add a partial index on `verification_failure` and `verification_checksum` to ensure - re-verification can be performed efficiently: +1. Add indexes on verification fields to ensure verification can be performed efficiently: + + Some or all of these indexes can be omitted if the table is guaranteed to be small. Ask a database expert if you are unsure. ```ruby # frozen_string_literal: true - class AddVerificationFailureIndexToWidgets < ActiveRecord::Migration[6.0] + class AddVerificationIndexesToWidgets < ActiveRecord::Migration[6.0] include Gitlab::Database::MigrationHelpers DOWNTIME = false + VERIFICATION_STATE_INDEX_NAME = "index_widgets_verification_state" + PENDING_VERIFICATION_INDEX_NAME = "index_widgets_pending_verification" + FAILED_VERIFICATION_INDEX_NAME = "index_widgets_failed_verification" + NEEDS_VERIFICATION_INDEX_NAME = "index_widgets_needs_verification" disable_ddl_transaction! def up - add_concurrent_index :widgets, :verification_failure, where: "(verification_failure IS NOT NULL)", name: "widgets_verification_failure_partial" - add_concurrent_index :widgets, :verification_checksum, where: "(verification_checksum IS NOT NULL)", name: "widgets_verification_checksum_partial" + add_concurrent_index :widgets, :verification_state, name: VERIFICATION_STATE_INDEX_NAME + add_concurrent_index :widgets, :verified_at, where: "(verification_state = 0)", order: { verified_at: 'ASC NULLS FIRST' }, name: PENDING_VERIFICATION_INDEX_NAME + add_concurrent_index :widgets, :verification_retry_at, where: "(verification_state = 3)", order: { verification_retry_at: 'ASC NULLS FIRST' }, name: FAILED_VERIFICATION_INDEX_NAME + add_concurrent_index :widgets, :verification_state, where: "(verification_state = 0 OR verification_state = 3)", name: NEEDS_VERIFICATION_INDEX_NAME end def down - remove_concurrent_index :widgets, :verification_failure - remove_concurrent_index :widgets, :verification_checksum + remove_concurrent_index_by_name :widgets, VERIFICATION_STATE_INDEX_NAME + remove_concurrent_index_by_name :widgets, PENDING_VERIFICATION_INDEX_NAME + remove_concurrent_index_by_name :widgets, FAILED_VERIFICATION_INDEX_NAME + remove_concurrent_index_by_name :widgets, NEEDS_VERIFICATION_INDEX_NAME end end ``` +1. Add the `Gitlab::Geo::VerificationState` concern to the `widget` model if it is not already included in `Gitlab::Geo::ReplicableModel`: + + ```ruby + class Widget < ApplicationRecord + ... + include ::Gitlab::Geo::VerificationState + ... + end + ``` + ##### Option 2: Create a separate `widget_states` table with verification state fields -1. Create a `widget_states` table and add a partial index on `verification_failure` and - `verification_checksum` to ensure re-verification can be performed efficiently. Order - the columns according to [the guidelines](../ordering_table_columns.md): +1. Create a `widget_states` table and add an index on `verification_state` to ensure verification can be performed efficiently. Order the columns according to [the guidelines](../ordering_table_columns.md): ```ruby # frozen_string_literal: true @@ -477,14 +496,15 @@ can track verification state. with_lock_retries do create_table :widget_states, id: false do |t| t.references :widget, primary_key: true, null: false, foreign_key: { on_delete: :cascade } + t.integer :verification_state, default: 0, limit: 2, null: false + t.column :verification_started_at, :datetime_with_timezone t.datetime_with_timezone :verification_retry_at t.datetime_with_timezone :verified_at t.integer :verification_retry_count, limit: 2 t.binary :verification_checksum, using: 'verification_checksum::bytea' t.text :verification_failure - t.index :verification_failure, where: "(verification_failure IS NOT NULL)", name: "widgets_verification_failure_partial" - t.index :verification_checksum, where: "(verification_checksum IS NOT NULL)", name: "widgets_verification_checksum_partial" + t.index :verification_state, name: "index_widget_states_on_verification_state" end end end @@ -498,6 +518,20 @@ can track verification state. end ``` +1. Add the following lines to the `widget_state.rb` model: + + ```ruby + class WidgetState < ApplicationRecord + ... + self.primary_key = :widget_id + + include ::Gitlab::Geo::VerificationState + + belongs_to :widget, inverse_of: :widget_state + ... + end + ``` + 1. Add the following lines to the `widget` model: ```ruby @@ -547,14 +581,16 @@ Metrics are gathered by `Geo::MetricsUpdateWorker`, persisted in 1. Add the following to `spec/factories/widgets.rb`: ```ruby - trait(:checksummed) do + trait(:verification_succeeded) do with_file verification_checksum { 'abc' } + verification_state { Widget.verification_state_value(:verification_succeeded) } end - trait(:checksum_failure) do + trait(:verification_failed) do with_file verification_failure { 'Could not calculate the checksum' } + verification_state { Widget.verification_state_value(:verification_failed) } end ``` |