Merge branch 'use-optimistic-locking' into 'master'

Use optimistic locking

## What does this MR do?
Removes the usage of pessimistic locking in favor of optimistic which is way cheaper and doesn't block database operation.

Since this is very simple change it should be safe. If we receive `StaleObjectError` message we will reload object a retry operations in lock.

However, I still believe that we need this one: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7005 as this will reduce a load on Database and FS.
This changes a behavior from:

### Pesimistic locking (previous behavior)

#### For updating
1. SELECT * FOR UPDATE (other updates wait on this)
2. we update ci_pipeline
3. latest_build_status
4. enqueue: (use: transition :created -> :pending)
5. [state_machine] we are in  state created, we can go to pending
6. [state_machine] ci_pipeline.status = created
7. [state_machine] ci_pipeline.save
8. [state_machine] after_transition: (if for success): PipelineSuccessWorker on Sidekiq
9. release DB lock

#### If no update is required
1. SELECT * FOR UPDATE (other updates wait on this)
2. we update ci_pipeline
3. latest_build_status
4. we are in pending, we can't transition to pending, because it's forbidden
5. release DB lock

### Optimistic locking (implemented by this MR)

#### For updating
1. latest_build_status
2. enqueue: (use `transition :created -> :pending`)
3. [state_machine] we are in state created, we can go to pending
4. [state_machine] ci_pipeline.status = created
5. [state_machine] ci_pipeline.save
6. [state_machine] [save] where(lock_version: ci_pipeline.lock_version).update_all(status: :created, updated_at: Time.now)
7. [state_machine] [save] unless we_updated_row then raise ObjectInconsistentError

#### If no update is required
1. we update ci_pipeline
2. latest_build_status
3. we are in pending, we can't transition to pending, because it's forbidden

## Why was this MR needed?
We have been seeing a number of problems when we migrated Pipeline/Build processing to Sidekiq. Especially we started seeing a lot of blocking queries.

We used a pessimistic locking which doesn't seem to be required. This effectively allows us to fix our issues with blocked queries by using more efficient method of operation.

## What are the relevant issue numbers?
Issues: https://gitlab.com/gitlab-com/infrastructure/issues/623 and https://gitlab.com/gitlab-com/infrastructure/issues/584, but also there's a bunch of Merge Requests that try to improve behavior of scheduled jobs.

cc @pcarranza @yorickpeterse @stanhu

See merge request !7040

2 files changed, 41 insertions, 0 deletions

diff --git a/spec/lib/gitlab/import_export/safe_model_attributes.yml b/spec/lib/gitlab/import_export/safe_model_attributes.yml
index feee0f025d8..07a2c316899 100644
--- a/spec/lib/gitlab/import_export/safe_model_attributes.yml
+++ b/spec/lib/gitlab/import_export/safe_model_attributes.yml
@@ -178,6 +178,7 @@ Ci::Pipeline:
 - finished_at
 - duration
 - user_id
+- lock_version
 CommitStatus:
 - id
 - project_id
@@ -217,6 +218,7 @@ CommitStatus:
 - yaml_variables
 - queued_at
 - token
+- lock_version
 Ci::Variable:
 - id
 - project_id
diff --git a/spec/lib/gitlab/optimistic_locking_spec.rb b/spec/lib/gitlab/optimistic_locking_spec.rb
new file mode 100644
index 00000000000..498dc514c8c
--- /dev/null
+++ b/spec/lib/gitlab/optimistic_locking_spec.rb
@@ -0,0 +1,39 @@
+require 'spec_helper'
+
+describe Gitlab::OptimisticLocking, lib: true do
+  describe '#retry_lock' do
+    let!(:pipeline) { create(:ci_pipeline) }
+    let!(:pipeline2) { Ci::Pipeline.find(pipeline.id) }
+
+    it 'does not reload object if state changes' do
+      expect(pipeline).not_to receive(:reload)
+      expect(pipeline).to receive(:succeed).and_call_original
+
+      described_class.retry_lock(pipeline) do |subject|
+        subject.succeed
+      end
+    end
+
+    it 'retries action if exception is raised' do
+      pipeline.succeed
+
+      expect(pipeline2).to receive(:reload).and_call_original
+      expect(pipeline2).to receive(:drop).twice.and_call_original
+
+      described_class.retry_lock(pipeline2) do |subject|
+        subject.drop
+      end
+    end
+
+    it 'raises exception when too many retries' do
+      expect(pipeline).to receive(:drop).twice.and_call_original
+
+      expect do
+        described_class.retry_lock(pipeline, 1) do |subject|
+          subject.lock_version = 100
+          subject.drop
+        end
+      end.to raise_error(ActiveRecord::StaleObjectError)
+    end
+  end
+end