Backport the EE schema and migrations to CE

This backports all EE schema changes to CE, including EE migrations,
ensuring both use the same schema.

== Updated tests

A spec related to ghost and support bot users had to be modified to make
it pass. The spec in question assumes that the "support_bot" column
exists when defining the spec. In the single codebase setup this is not
the case, as the column is backported in a later migration. Any attempt
to use a different schema version or use of "around" blocks to
conditionally disable specs won't help, as reverting the backport
migration would also drop the "support_bot" column. Removing the
"support_bot" tests entirely appears to be the only solution.

We also need to update some foreign key tests now that we have
backported the EE columns. Fortunately, these changes are very minor.

== Backporting migrations

This commit moves EE specific migrations (except those for the Geo
tracking database) and related files to CE, and also removes any traces
of the ee/db directory.

Some migrations had to be modified or removed, as they no longer work
with the schema being backported. These migrations were all quite old,
so we opted for removing them where modifying them would take too much
time and effort.

Some old migrations were modified in EE, while also existing in CE. In
these cases we took the EE code, and in one case removed them entirely.
It's not worth spending time trying to merge these changes somehow as we
plan to remove old migrations around the release of 12.0, see
https://gitlab.com/gitlab-org/gitlab-ce/issues/59177 for more details.

3 files changed, 108 insertions, 7 deletions

diff --git a/lib/gitlab/import_export/config.rb b/lib/gitlab/import_export/config.rb
new file mode 100644
index 00000000000..f6cd4eb5e0c
--- /dev/null
+++ b/lib/gitlab/import_export/config.rb
@@ -0,0 +1,78 @@
+# frozen_string_literal: true
+
+module Gitlab
+  module ImportExport
+    class Config
+      # Returns a Hash of the YAML file, including EE specific data if EE is
+      # used.
+      def to_h
+        hash = parse_yaml
+        ee_hash = hash['ee']
+
+        if merge? && ee_hash
+          ee_hash.each do |key, value|
+            if key == 'project_tree'
+              merge_project_tree(value, hash[key])
+            else
+              merge_attributes_list(value, hash[key])
+            end
+          end
+        end
+
+        # We don't want to expose this section after this point, as it is no
+        # longer needed.
+        hash.delete('ee')
+
+        hash
+      end
+
+      # Merges a project relationships tree into the target tree.
+      #
+      # @param [Array<Hash|Symbol>] source_values
+      # @param [Array<Hash|Symbol>] target_values
+      def merge_project_tree(source_values, target_values)
+        source_values.each do |value|
+          if value.is_a?(Hash)
+            # Examples:
+            #
+            # { 'project_tree' => [{ 'labels' => [...] }] }
+            # { 'notes' => [:author, { 'events' => [:push_event_payload] }] }
+            value.each do |key, val|
+              target = target_values
+                .find { |h| h.is_a?(Hash) && h[key] }
+
+              if target
+                merge_project_tree(val, target[key])
+              else
+                target_values << { key => val.dup }
+              end
+            end
+          else
+            # Example: :priorities, :author, etc
+            target_values << value
+          end
+        end
+      end
+
+      # Merges a Hash containing a flat list of attributes, such as the entries
+      # in a `excluded_attributes` section.
+      #
+      # @param [Hash] source_values
+      # @param [Hash] target_values
+      def merge_attributes_list(source_values, target_values)
+        source_values.each do |key, values|
+          target_values[key] ||= []
+          target_values[key].concat(values)
+        end
+      end
+
+      def merge?
+        Gitlab.ee?
+      end
+
+      def parse_yaml
+        YAML.load_file(Gitlab::ImportExport.config_file)
+      end
+    end
+  end
+end
diff --git a/lib/gitlab/import_export/import_export.yml b/lib/gitlab/import_export/import_export.yml
index 71c44af9254..a0fb051e806 100644
--- a/lib/gitlab/import_export/import_export.yml
+++ b/lib/gitlab/import_export/import_export.yml
@@ -1,7 +1,11 @@
 # Model relationships to be included in the project import/export
+#
+# This list _must_ only contain relationships that are available to both CE and
+# EE. EE specific relationships must be defined in the `ee` section further
+# down below.
 project_tree:
   - labels:
-      :priorities
+    - :priorities
   - milestones:
     - events:
       - :push_event_payload
@@ -15,18 +19,18 @@ project_tree:
         - :push_event_payload
     - label_links:
       - label:
-          :priorities
+        - :priorities
     - milestone:
       - events:
         - :push_event_payload
     - resource_label_events:
       - label:
-          :priorities
+        - :priorities
     - :issue_assignees
   - snippets:
     - :award_emoji
     - notes:
-        :author
+      - :author
   - releases:
     - :links
   - project_members:
@@ -46,13 +50,13 @@ project_tree:
     - :timelogs
     - label_links:
       - label:
-          :priorities
+        - :priorities
     - milestone:
       - events:
         - :push_event_payload
     - resource_label_events:
       - label:
-          :priorities
+        - :priorities
   - ci_pipelines:
     - notes:
       - :author
@@ -121,12 +125,22 @@ excluded_attributes:
     - :bfg_object_map
     - :detected_repository_languages
     - :tag_list
+    - :mirror_user_id
+    - :mirror_trigger_builds
+    - :only_mirror_protected_branches
+    - :pull_mirror_available_overridden
+    - :mirror_overwrites_diverged_branches
+    - :packages_enabled
+    - :mirror_last_update_at
+    - :mirror_last_successful_update_at
   namespaces:
     - :runners_token
     - :runners_token_encrypted
   project_import_state:
     - :last_error
     - :jid
+    - :last_update_at
+    - :last_successful_update_at
   prometheus_metrics:
     - :common
     - :identifier
@@ -201,3 +215,12 @@ methods:
     - :action
   project_badges:
     - :type
+
+# EE specific relationships and settings to include. All of this will be merged
+# into the previous structures if EE is used.
+ee:
+  project_tree:
+    - protected_branches:
+      - :unprotect_access_levels
+    - protected_environments:
+      - :deploy_access_levels
diff --git a/lib/gitlab/import_export/reader.rb b/lib/gitlab/import_export/reader.rb
index bc0d18e03fa..8bdf6ca491d 100644
--- a/lib/gitlab/import_export/reader.rb
+++ b/lib/gitlab/import_export/reader.rb
@@ -7,7 +7,7 @@ module Gitlab
 
       def initialize(shared:)
         @shared = shared
-        config_hash = YAML.load_file(Gitlab::ImportExport.config_file).deep_symbolize_keys
+        config_hash = ImportExport::Config.new.to_h.deep_symbolize_keys
         @tree = config_hash[:project_tree]
         @attributes_finder = Gitlab::ImportExport::AttributesFinder.new(included_attributes: config_hash[:included_attributes],
                                                                         excluded_attributes: config_hash[:excluded_attributes],