summaryrefslogtreecommitdiff
path: root/doc/development/internal_api.md
blob: e25feda356d62a4e7d1be6af9853358490cc4e1c (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
---
stage: Create
group: Source Code
info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers"
type: reference, api
---

# Internal API

The internal API is used by different GitLab components, it can not be
used by other consumers. This documentation is intended for people
working on the GitLab codebase.

This documentation does not yet include the internal API used by
GitLab Pages.

## Authentication

These methods are all authenticated using a shared secret. This secret
is stored in a file at the path configured in `config/gitlab.yml` by
default this is in the root of the rails app named
`.gitlab_shell_secret`

To authenticate using that token, clients read the contents of that
file, and include the token Base64 encoded in a `secret_token` parameter
or in the `Gitlab-Shared-Secret` header.

NOTE: **Note:**
The internal API used by GitLab Pages, and GitLab Kubernetes Agent Server (kas) uses JSON Web Token (JWT)
authentication, which is different from GitLab Shell.

## Git Authentication

This is called by [Gitaly](https://gitlab.com/gitlab-org/gitaly) and
[GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) to check access to a
repository.

When called from GitLab Shell no changes are passed and the internal
API replies with the information needed to pass the request on to
Gitaly.

When called from Gitaly in a `pre-receive` hook the changes are passed
and those are validated to determine if the push is allowed.

```plaintext
POST /internal/allowed
```

| Attribute | Type   | Required | Description |
|:----------|:-------|:---------|:------------|
| `key_id`  | string | no       | ID of the SSH-key used to connect to GitLab Shell |
| `username` | string | no      | Username from the certificate used to connect to GitLab Shell |
| `project`  | string | no (if `gl_repository` is passed) | Path to the project |
| `gl_repository`  | string | no (if `project` is passed) | Repository identifier (e.g. `project-7`) |
| `protocol` | string | yes     | SSH when called from GitLab Shell, HTTP or SSH when called from Gitaly |
| `action`   | string | yes     | Git command being run (`git-upload-pack`, `git-receive-pack`, `git-upload-archive`) |
| `changes`  | string | yes     | `<oldrev> <newrev> <refname>` when called from Gitaly, the magic string `_any` when called from GitLab Shell |
| `check_ip` | string | no     | IP address from which call to GitLab Shell was made |

Example request:

```shell
curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" --data "key_id=11&project=gnuwget/wget2&action=git-upload-pack&protocol=ssh" http://localhost:3001/api/v4/internal/allowed
```

Example response:

```json
{
  "status": true,
  "gl_repository": "project-3",
  "gl_project_path": "gnuwget/wget2",
  "gl_id": "user-1",
  "gl_username": "root",
  "git_config_options": [],
  "gitaly": {
    "repository": {
      "storage_name": "default",
      "relative_path": "@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.git",
      "git_object_directory": "",
      "git_alternate_object_directories": [],
      "gl_repository": "project-3",
      "gl_project_path": "gnuwget/wget2"
    },
    "address": "unix:/Users/bvl/repos/gitlab/gitaly.socket",
    "token": null
  },
  "gl_console_messages": []
}
```

### Known consumers

- Gitaly
- GitLab Shell

## LFS Authentication

This is the endpoint that gets called from GitLab Shell to provide
information for LFS clients when the repository is accessed over SSH.

| Attribute | Type   | Required | Description |
|:----------|:-------|:---------|:------------|
| `key_id`  | string | no       | ID of the SSH-key used to connect to GitLab Shell |
| `username`| string | no       | Username from the certificate used to connect to GitLab Shell |
| `project` | string | no       | Path to the project |

Example request:

```shell
curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" --data "key_id=11&project=gnuwget/wget2" http://localhost:3001/api/v4/internal/lfs_authenticate
```

```json
{
  "username": "root",
  "lfs_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJkYXRhIjp7ImFjdG9yIjoicm9vdCJ9LCJqdGkiOiIyYWJhZDcxZC0xNDFlLTQ2NGUtOTZlMi1mODllYWRiMGVmZTYiLCJpYXQiOjE1NzAxMTc2NzYsIm5iZiI6MTU3MDExNzY3MSwiZXhwIjoxNTcwMTE5NDc2fQ.g7atlBw1QMY7QEBVPE0LZ8ZlKtaRzaMRmNn41r2YITM",
  "repository_http_path": "http://localhost:3001/gnuwget/wget2.git",
  "expires_in": 1800
}
```

### Known consumers

- GitLab Shell

## Authorized Keys Check

This endpoint is called by the GitLab Shell authorized keys
check. Which is called by OpenSSH for [fast SSH key
lookup](../administration/operations/fast_ssh_key_lookup.md).

| Attribute | Type   | Required | Description |
|:----------|:-------|:---------|:------------|
| `key`     | string | yes      | SSH key as passed by OpenSSH to GitLab Shell |

```plaintext
GET /internal/authorized_keys
```

Example request:

```shell
curl --request GET --header "Gitlab-Shared-Secret: <Base64 encoded secret>""http://localhost:3001/api/v4/internal/authorized_keys?key=<key as passed by OpenSSH>"
```

Example response:

```json
{
  "id": 11,
  "title": "admin@example.com",
  "key": "ssh-rsa ...",
  "created_at": "2019-06-27T15:29:02.219Z"
}
```

### Known consumers

- GitLab Shell

## Get user for user ID or key

This endpoint is used when a user performs `ssh git@gitlab.com`. It
discovers the user associated with an SSH key.

| Attribute | Type   | Required | Description |
|:----------|:-------|:---------|:------------|
| `key_id` | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check |
| `username` | string | no | Username of the user being looked up, used by GitLab Shell when authenticating using a certificate |

```plaintext
GET /internal/discover
```

Example request:

```shell
curl --request GET --header "Gitlab-Shared-Secret: <Base64 encoded secret>" "http://localhost:3001/api/v4/internal/discover?key_id=7"
```

Example response:

```json
{
  "id": 7,
  "name": "Dede Eichmann",
  "username": "rubi"
}
```

### Known consumers

- GitLab Shell

## Instance information

This gets some generic information about the instance. This is used
by Geo nodes to get information about each other.

```plaintext
GET /internal/check
```

Example request:

```shell
curl --request GET --header "Gitlab-Shared-Secret: <Base64 encoded secret>" "http://localhost:3001/api/v4/internal/check"
```

Example response:

```json
{
  "api_version": "v4",
  "gitlab_version": "12.3.0-pre",
  "gitlab_rev": "d69c988e6a6",
  "redis": true
}
```

### Known consumers

- GitLab Geo
- GitLab Shell's `bin/check`
- Gitaly

## Get new 2FA recovery codes using an SSH key

This is called from GitLab Shell and allows users to get new 2FA
recovery codes based on their SSH key.

| Attribute | Type   | Required | Description |
|:----------|:-------|:---------|:------------|
| `key_id`  | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check |
| `user_id` | integer | no | **Deprecated** User_id for which to generate new recovery codes |

```plaintext
GET /internal/two_factor_recovery_codes
```

Example request:

```shell
curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" --data "key_id=7" http://localhost:3001/api/v4/internal/two_factor_recovery_codes
```

Example response:

```json
{
  "success": true,
  "recovery_codes": [
    "d93ee7037944afd5",
    "19d7b84862de93dd",
    "1e8c52169195bf71",
    "be50444dddb7ca84",
    "26048c77d161d5b7",
    "482d5c03d1628c47",
    "d2c695e309ce7679",
    "dfb4748afc4f12a7",
    "0e5f53d1399d7979",
    "af04d5622153b020"
  ]
}
```

### Known consumers

- GitLab Shell

## Get new personal access-token

This is called from GitLab Shell and allows users to generate a new
personal access token.

| Attribute | Type   | Required | Description |
|:----------|:-------|:---------|:------------|
| `name` | string | yes | The name of the new token |
| `scopes` | string array | yes | The authorization scopes for the new token, these must be valid token scopes |
| `expires_at` | string | no | The expiry date for the new token |
| `key_id`  | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check |
| `user_id` | integer | no | User\_id for which to generate the new token |

```plaintext
POST /internal/personal_access_token
```

Example request:

```shell
curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" --data "user_id=29&name=mytokenname&scopes[]=read_user&scopes[]=read_repository&expires_at=2020-07-24" http://localhost:3001/api/v4/internal/personal_access_token
```

Example response:

```json
{
  "success": true,
  "token": "Hf_79B288hRv_3-TSD1R",
  "scopes": ["read_user","read_repository"],
  "expires_at": "2020-07-24"
}
```

### Known consumers

- GitLab Shell

## Incrementing counter on pre-receive

This is called from the Gitaly hooks increasing the reference counter
for a push that might be accepted.

| Attribute | Type   | Required | Description |
|:----------|:-------|:---------|:------------|
| `gl_repository` | string | yes | repository identifier for the repository receiving the push |

```plaintext
POST /internal/pre_receive
```

Example request:

```shell
curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" --data "gl_repository=project-7" http://localhost:3001/api/v4/internal/pre_receive
```

Example response:

```json
{
  "reference_counter_increased": true
}
```

## PostReceive

Called from Gitaly after a receiving a push. This triggers the
`PostReceive`-worker in Sidekiq, processes the passed push options and
builds the response including messages that need to be displayed to
the user.

| Attribute | Type   | Required | Description |
|:----------|:-------|:---------|:------------|
| `identifier` | string | yes | `user-[id]` or `key-[id]` Identifying the user performing the push |
| `gl_repository` | string | yes | identifier of the repository being pushed to |
| `push_options` | string array | no | array of push options |
| `changes` | string | no | refs to be updated in the push in the format `oldrev newrev refname\n`. |

```plaintext
POST /internal/post_receive
```

Example Request:

```shell
curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" --data "gl_repository=project-7" --data "identifier=user-1" --data "changes=0000000000000000000000000000000000000000 fd9e76b9136bdd9fe217061b497745792fe5a5ee gh-pages\n"  http://localhost:3001/api/v4/internal/post_receive
```

Example response:

```json
{
  "messages": [
    {
      "message": "Hello from post-receive",
      "type": "alert"
    }
  ],
  "reference_counter_decreased": true
}
```

## Kubernetes agent endpoints

> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41045) in GitLab 13.4.
> - This feature is not deployed on GitLab.com
> - It's not recommended for production use.

The following endpoints are used by the GitLab Kubernetes Agent Server (kas)
for various purposes.

These endpoints are all authenticated using JWT. The JWT secret is stored in a file
specified in `config/gitlab.yml`. By default, the location is in the root of the
GitLab Rails app in a file called `.gitlab_kas_secret`.

CAUTION: **Caution:**
The Kubernetes agent is under development and is not recommended for production use.

### Kubernetes agent information

Called from GitLab Kubernetes Agent Server (kas) to retrieve agent
information for the given agent token. This returns the Gitaly connection
information for the agent's project in order for kas to fetch and update
the agent's configuration.

```plaintext
GET /internal/kubernetes/agent_info
```

Example Request:

```shell
curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/agent_info"
```

### Kubernetes agent project information

Called from GitLab Kubernetes Agent Server (kas) to retrieve project
information for the given agent token. This returns the Gitaly
connection for the requested project. GitLab kas uses this to configure
the agent to fetch Kubernetes resources from the project repository to
sync.

Only public projects are currently supported. For private projects, the ability for the
agent to be authorized is [not yet implemented](https://gitlab.com/gitlab-org/gitlab/-/issues/220912).

| Attribute | Type   | Required | Description |
|:----------|:-------|:---------|:------------|
| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../api/README.md#namespaced-path-encoding) |

```plaintext
GET /internal/kubernetes/project_info
```

Example Request:

```shell
curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/project_info?id=7"
```

### Kubernetes agent usage metrics

Called from GitLab Kubernetes Agent Server (kas) to increase the usage
metric counters.

| Attribute | Type   | Required | Description |
|:----------|:-------|:---------|:------------|
| `gitops_sync_count` | integer| yes | The number to increase the `gitops_sync_count` counter by |

```plaintext
POST /internal/kubernetes/usage_metrics
```

Example Request:

```shell
curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Content-Type: application/json" --data '{"gitops_sync_count":1}' "http://localhost:3000/api/v4/internal/kubernetes/usage_metrics"
```