summaryrefslogtreecommitdiff
path: root/doc/development/import_export.md
blob: 6d1b6929667b0c6f7387b6da7afe5bafb5e3f9bb (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
# Import/Export development documentation

Troubleshooting and general development guidelines and tips for the [Import/Export feature](../user/project/settings/import_export.md).

<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> This document is originally based on the [Import/Export 201 presentation available on YouTube](https://www.youtube.com/watch?v=V3i1OfExotE).

## Troubleshooting commands

Finds information about the status of the import and further logs using the JID:

```ruby
# Rails console
Project.find_by_full_path('group/project').import_state.slice(:jid, :status, :last_error)
> {"jid"=>"414dec93f941a593ea1a6894", "status"=>"finished", "last_error"=>nil}
```

```shell
# Logs
grep JID /var/log/gitlab/sidekiq/current
grep "Import/Export error" /var/log/gitlab/sidekiq/current
grep "Import/Export backtrace" /var/log/gitlab/sidekiq/current
tail /var/log/gitlab/gitlab-rails/importer.log
```

## Troubleshooting performance issues

Read through the current performance problems using the Import/Export below.

### OOM errors

Out of memory (OOM) errors are normally caused by the [Sidekiq Memory Killer](../administration/operations/sidekiq_memory_killer.md):

```shell
SIDEKIQ_MEMORY_KILLER_MAX_RSS = 2000000
SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS = 3000000
SIDEKIQ_MEMORY_KILLER_GRACE_TIME = 900
```

An import status `started`, and the following Sidekiq logs will signal a memory issue:

```shell
WARN: Work still in progress <struct with JID>
```

### Timeouts

Timeout errors occur due to the `StuckImportJobsWorker` marking the process as failed:

```ruby
class StuckImportJobsWorker
  include ApplicationWorker
  include CronjobQueue

  IMPORT_JOBS_EXPIRATION = 15.hours.to_i

  def perform
    import_state_without_jid_count = mark_import_states_without_jid_as_failed!
    import_state_with_jid_count = mark_import_states_with_jid_as_failed!
    ...
```

```shell
Marked stuck import jobs as failed. JIDs: xyz
```

```plaintext
  +-----------+    +-----------------------------------+
  |Export Job |--->| Calls ActiveRecord `as_json` and  |
  +-----------+    | `to_json` on all project models   |
                   +-----------------------------------+

  +-----------+    +-----------------------------------+
  |Import Job |--->| Loads all JSON in memory, then    |
  +-----------+    | inserts into the DB in batches    |
                   +-----------------------------------+
```

### Problems and solutions

| Problem | Possible solutions |
| -------- | -------- |
| [Slow JSON](https://gitlab.com/gitlab-org/gitlab/-/issues/25251) loading/dumping models from the database | [split the worker](https://gitlab.com/gitlab-org/gitlab/-/issues/25252) |
| | Batch export
| | Optimize SQL
| | Move away from `ActiveRecord` callbacks (difficult)
| High memory usage (see also some [analysis](https://gitlab.com/gitlab-org/gitlab/-/issues/18857) | DB Commit sweet spot that uses less memory |
| | [Netflix Fast JSON API](https://github.com/Netflix/fast_jsonapi) may help |
| | Batch reading/writing to disk and any SQL

### Temporary solutions

While the performance problems are not tackled, there is a process to workaround
importing big projects, using a foreground import:

[Foreground import](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5384) of big projects for customers.
(Using the import template in the [infrastructure tracker](https://gitlab.com/gitlab-com/gl-infra/infrastructure/))

## Security

The Import/Export feature is constantly updated (adding new things to export), however
the code hasn't been refactored in a long time. We should perform a code audit (see
[confidential issue](../user/project/issues/confidential_issues.md) `https://gitlab.com/gitlab-org/gitlab/issues/20720`).
to make sure its dynamic nature does not increase the number of security concerns.

### Security in the code

Some of these classes provide a layer of security to the Import/Export.

The `AttributeCleaner` removes any prohibited keys:

```ruby
# AttributeCleaner
# Removes all `_ids` and other prohibited keys
    class AttributeCleaner
      ALLOWED_REFERENCES = RelationFactory::PROJECT_REFERENCES + RelationFactory::USER_REFERENCES + ['group_id']

      def clean
        @relation_hash.reject do |key, _value|
          prohibited_key?(key) || !@relation_class.attribute_method?(key) || excluded_key?(key)
        end.except('id')
      end

      ...

```

The `AttributeConfigurationSpec` checks and confirms the addition of new columns:

```ruby
# AttributeConfigurationSpec
<<-MSG
  It looks like #{relation_class}, which is exported using the project Import/Export, has new attributes:

  Please add the attribute(s) to SAFE_MODEL_ATTRIBUTES if you consider this can be exported.
  Otherwise, please blacklist the attribute(s) in IMPORT_EXPORT_CONFIG by adding it to its correspondent
  model in the +excluded_attributes+ section.

  SAFE_MODEL_ATTRIBUTES: #{File.expand_path(safe_attributes_file)}
  IMPORT_EXPORT_CONFIG: #{Gitlab::ImportExport.config_file}
MSG
```

The `ModelConfigurationSpec` checks and confirms the addition of new models:

```ruby
# ModelConfigurationSpec
<<-MSG
  New model(s) <#{new_models.join(',')}> have been added, related to #{parent_model_name}, which is exported by
  the Import/Export feature.

  If you think this model should be included in the export, please add it to `#{Gitlab::ImportExport.config_file}`.

  Definitely add it to `#{File.expand_path(ce_models_yml)}`
  to signal that you've handled this error and to prevent it from showing up in the future.
MSG
```

The `ExportFileSpec` detects encrypted or sensitive columns:

```ruby
# ExportFileSpec
<<-MSG
  Found a new sensitive word <#{key_found}>, which is part of the hash #{parent.inspect}
  If you think this information shouldn't get exported, please exclude the model or attribute in
  IMPORT_EXPORT_CONFIG.

  Otherwise, please add the exception to +safe_list+ in CURRENT_SPEC using #{sensitive_word} as the
  key and the correspondent hash or model as the value.

  Also, if the attribute is a generated unique token, please add it to RelationFactory::TOKEN_RESET_MODELS
  if it needs to be reset (to prevent duplicate column problems while importing to the same instance).

  IMPORT_EXPORT_CONFIG: #{Gitlab::ImportExport.config_file}
  CURRENT_SPEC: #{__FILE__}
MSG
```

## Versioning

Import/Export does not use strict SemVer, since it has frequent constant changes
during a single GitLab release. It does require an update when there is a breaking change.

```ruby
# ImportExport
module Gitlab
  module ImportExport
    extend self

    # For every version update, the version history in import_export.md has to be kept up to date.
    VERSION = '0.2.4'
```

## Version history

The [current version history](../user/project/settings/import_export.md) also displays the equivalent GitLab version
and it is useful for knowing which versions won't be compatible between them.

| Exporting GitLab version   | Importing GitLab version   |
| -------------------------- | -------------------------- |
| 11.7 to current            | 11.7 to current            |
| 11.1 to 11.6               | 11.1 to 11.6               |
| 10.8 to 11.0               | 10.8 to 11.0               |
| 10.4 to 10.7               | 10.4 to 10.7               |
| ...                        | ...                        |
| 8.10.3 to 8.11             | 8.10.3 to 8.11             |
| 8.10.0 to 8.10.2           | 8.10.0 to 8.10.2           |
| 8.9.5 to 8.9.11            | 8.9.5 to 8.9.11            |
| 8.9.0 to 8.9.4             | 8.9.0 to 8.9.4             |

### When to bump the version up

We will have to bump the version if we rename model/columns or perform any format
modifications in the JSON structure or the file structure of the archive file.

We do not need to bump the version up in any of the following cases:

- Add a new column or a model
- Remove a column or model (unless there is a DB constraint)
- Export new things (such as a new type of upload)

Every time we bump the version, the integration specs will fail and can be fixed with:

```shell
bundle exec rake gitlab:import_export:bump_version
```

## A quick dive into the code

### Import/Export configuration (`import_export.yml`)

The main configuration `import_export.yml` defines what models can be exported/imported.

Model relationships to be included in the project import/export:

```yaml
project_tree:
  - labels:
    - :priorities
  - milestones:
    - events:
      - :push_event_payload
  - issues:
    - events:
    - ...
```

Only include the following attributes for the models specified:

```yaml
included_attributes:
  user:
    - :id
    - :email
  ...

```

Do not include the following attributes for the models specified:

```yaml
excluded_attributes:
  project:
    - :name
    - :path
    - ...
```

Extra methods to be called by the export:

```yaml
# Methods
methods:
  labels:
    - :type
  label:
    - :type
```

### Import

The import job status moves from `none` to `finished` or `failed` into different states:

_import\_status_: none -> scheduled -> started -> finished/failed

While the status is `started` the `Importer` code processes each step required for the import.

```ruby
# ImportExport::Importer
module Gitlab
  module ImportExport
    class Importer
      def execute
        if import_file && check_version! && restorers.all?(&:restore) && overwrite_project
          project
        else
          raise Projects::ImportService::Error.new(@shared.errors.join(', '))
        end
      rescue => e
        raise Projects::ImportService::Error.new(e.message)
      ensure
        remove_import_file
      end

      def restorers
        [repo_restorer, wiki_restorer, project_tree, avatar_restorer,
         uploads_restorer, lfs_restorer, statistics_restorer]
      end
```

The export service, is similar to the `Importer`, restoring data instead of saving it.

### Export

```ruby
# ImportExport::ExportService
module Projects
  module ImportExport
    class ExportService < BaseService

      def save_all!
        if save_services
          Gitlab::ImportExport::Saver.save(project: project, shared: @shared)
          notify_success
        else
          cleanup_and_notify_error!
        end
      end

      def save_services
        [version_saver, avatar_saver, project_tree_saver, uploads_saver, repo_saver,
           wiki_repo_saver, lfs_saver].all?(&:save)
      end
```

## Test fixtures

Fixtures used in Import/Export specs live in `spec/fixtures/lib/gitlab/import_export`. There are both Project and Group fixtures.

There are two versions of each of these fixtures:

- A human readable single JSON file with all objects, called either `project.json` or `group.json`.
- A folder named `tree`, containing a tree of files in `ndjson` format. **Please do not edit files under this folder manually unless strictly necessary.**

The tools to generate the NDJSON tree from the human-readable JSON files live in the [`gitlab-org/memory-team/team-tools`](https://gitlab.com/gitlab-org/memory-team/team-tools/-/blob/master/import-export/) project.

### Project

**Please use `legacy-project-json-to-ndjson.sh` to generate the NDJSON tree.**

The NDJSON tree will look like this:

```shell
tree
├── project
│   ├── auto_devops.ndjson
│   ├── boards.ndjson
│   ├── ci_cd_settings.ndjson
│   ├── ci_pipelines.ndjson
│   ├── container_expiration_policy.ndjson
│   ├── custom_attributes.ndjson
│   ├── error_tracking_setting.ndjson
│   ├── external_pull_requests.ndjson
│   ├── issues.ndjson
│   ├── labels.ndjson
│   ├── merge_requests.ndjson
│   ├── milestones.ndjson
│   ├── pipeline_schedules.ndjson
│   ├── project_badges.ndjson
│   ├── project_feature.ndjson
│   ├── project_members.ndjson
│   ├── protected_branches.ndjson
│   ├── protected_tags.ndjson
│   ├── releases.ndjson
│   ├── services.ndjson
│   ├── snippets.ndjson
│   └── triggers.ndjson
└── project.json
```

### Group

**Please use `legacy-group-json-to-ndjson.rb` to generate the NDJSON tree.**

The NDJSON tree will look like this:

```shell
tree
└── groups
    ├── 4351
    │   ├── badges.ndjson
    │   ├── boards.ndjson
    │   ├── epics.ndjson
    │   ├── labels.ndjson
    │   ├── members.ndjson
    │   └── milestones.ndjson
    ├── 4352
    │   ├── badges.ndjson
    │   ├── boards.ndjson
    │   ├── epics.ndjson
    │   ├── labels.ndjson
    │   ├── members.ndjson
    │   └── milestones.ndjson
    ├── _all.ndjson
    ├── 4351.json
    └── 4352.json
```

CAUTION: **Caution:** When updating these fixtures, please ensure you update both `json` files and `tree` folder, as the tests apply to both.