1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
|
# Parsing GitLab logs with `jq`
We recommend using log aggregation and search tools like Kibana and Splunk whenever possible,
but if they are not available you can still quickly parse
[GitLab logs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26311) in JSON format
(the default since GitLab 12.0) using [`jq`](https://stedolan.github.io/jq/).
## What is JQ?
As noted in its [manual](https://stedolan.github.io/jq/manual/), jq is a command-line JSON processor. The following examples
include use cases targeted for parsing GitLab log files.
## Parsing Logs
### General Commands
#### Pipe colorized `jq` output into `less`
```sh
jq . <FILE> -C | less -R
```
#### Search for a term and pretty-print all matching lines
```sh
grep <TERM> <FILE> | jq .
```
#### Skip invalid lines of JSON
```sh
jq -cR 'fromjson?' file.json | jq <COMMAND>
```
By default `jq` will error out when it encounters a line that is not valid JSON.
This skips over all invalid lines and parses the rest.
### Parsing `production_json.log` and `api_json.log`
#### Find all requests with a 5XX status code
```sh
jq 'select(status >= 500)' <FILE>
```
#### Top 10 slowest requests
```sh
jq -s 'sort_by(-.duration) | limit(10; .[])' <FILE>
```
#### Find and pretty print all requests related to a project
```sh
grep <PROJECT_NAME> <FILE> | jq .
```
#### Find all requests with a total duration > 5 seconds
```sh
jq 'select(.duration > 5000)' <FILE>
```
#### Find all project requests with more than 5 rugged calls
```sh
grep <PROJECT_NAME> <FILE> | jq 'select(.rugged_calls > 5)'
```
#### Find all requests with a Gitaly duration > 10 seconds
```sh
jq 'select(.gitaly_duration > 10000)' <FILE>
```
#### Find all requests with a queue duration > 10 seconds
```sh
jq 'select(.queue_duration > 10000)' <FILE>
```
#### Top 10 requests by # of Gitaly calls
```sh
jq -s 'map(select(.gitaly_calls != null)) | sort_by(-.gitaly_calls) | limit(10; .[])' <FILE>
```
### Parsing `production_json.log`
#### Print the top three controller methods by request volume and their three longest durations
```sh
jq -s -r 'group_by(.controller+.action) | sort_by(-length) | limit(3; .[]) | sort_by(-.duration) | "CT: \(length)\tMETHOD: \(.[0].controller)#\(.[0].action)\tDURS: \(.[0].duration), \(.[1].duration), \(.[2].duration)"' production_json.log
```
**Example output**
```plaintext
CT: 2721 METHOD: SessionsController#new DURS: 844.06, 713.81, 704.66
CT: 2435 METHOD: MetricsController#index DURS: 299.29, 284.01, 158.57
CT: 1328 METHOD: Projects::NotesController#index DURS: 403.99, 386.29, 384.39
```
### Parsing `api_json.log`
#### Print top three routes with request count and their three longest durations
```sh
jq -s -r 'group_by(.route) | sort_by(-length) | limit(3; .[]) | sort_by(-.duration) | "CT: \(length)\tROUTE: \(.[0].route)\tDURS: \(.[0].duration), \(.[1].duration), \(.[2].duration)"' api_json.log
```
**Example output**
```plaintext
CT: 2472 ROUTE: /api/:version/internal/allowed DURS: 56402.65, 38411.43, 19500.41
CT: 297 ROUTE: /api/:version/projects/:id/repository/tags DURS: 731.39, 685.57, 480.86
CT: 190 ROUTE: /api/:version/projects/:id/repository/commits DURS: 1079.02, 979.68, 958.21
```
### Parsing `gitaly/current`
#### Find all Gitaly requests sent from web UI
```sh
jq 'select(."grpc.meta.client_name" == "gitlab-web")' current
```
#### Find all failed Gitaly requests
```sh
jq 'select(."grpc.code" != null and ."grpc.code" != "OK")' current
```
#### Find all requests that took longer than 30 seconds
```sh
jq 'select(."grpc.time_ms" > 30000)' current
```
#### Print top three projects by request volume and their three longest durations
```sh
jq -s -r 'map(select(."grpc.request.glProjectPath" != null and ."grpc.request.glProjectPath" != "" and ."grpc.time_ms" != null)) | group_by(."grpc.request.glProjectPath") | sort_by(-length) | limit(3; .[]) | sort_by(-."grpc.time_ms") | "CT: \(length)\tPROJECT: \(.[0]."grpc.request.glProjectPath")\tDURS: \(.[0]."grpc.time_ms"), \(.[1]."grpc.time_ms"), \(.[2]."grpc.time_ms")"' current
```
**Example output**
```plaintext
CT: 635 PROJECT: groupA/project1 DURS: 4292.269, 4228.853, 2885.548
CT: 462 PROJECT: groupB/project5 DURS: 4368.981, 3623.553, 361.399
CT: 455 PROJECT: groupC/project7 DURS: 387.295, 381.874, 373.988
```
|
