summaryrefslogtreecommitdiff
path: root/doc/administration/logs.md
diff options
context:
space:
mode:
Diffstat (limited to 'doc/administration/logs.md')
-rw-r--r--doc/administration/logs.md220
1 files changed, 191 insertions, 29 deletions
diff --git a/doc/administration/logs.md b/doc/administration/logs.md
index e4a3514a539..57b12897979 100644
--- a/doc/administration/logs.md
+++ b/doc/administration/logs.md
@@ -1,3 +1,9 @@
+---
+stage: Monitor
+group: APM
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# Log system
GitLab has an advanced log system where everything is logged, so you
@@ -333,6 +339,9 @@ only. For example:
## `audit_json.log`
+NOTE: **Note:**
+Most log entries only exist in [GitLab Starter](https://about.gitlab.com/pricing), however a few exist in GitLab Core.
+
This file lives in `/var/log/gitlab/gitlab-rails/audit_json.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/audit_json.log` for
installations from source.
@@ -356,7 +365,11 @@ Changes to group or project settings are logged to this file. For example:
}
```
-## `sidekiq.log`
+## Sidekiq Logs
+
+For Omnibus installations, some Sidekiq logs reside in `/var/log/gitlab/sidekiq/current` and as follows.
+
+### `sidekiq.log`
This file lives in `/var/log/gitlab/gitlab-rails/sidekiq.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/sidekiq.log` for
@@ -413,7 +426,7 @@ For source installations, edit the `gitlab.yml` and set the Sidekiq
log_format: json
```
-## `sidekiq_client.log`
+### `sidekiq_client.log`
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26586) in GitLab 12.9.
@@ -424,21 +437,91 @@ installations from source.
This file contains logging information about jobs before they are start
being processed by Sidekiq, for example before being enqueued.
-This logfile follows the same structure as
+This log file follows the same structure as
[`sidekiq.log`](#sidekiqlog), so it will be structured as JSON if
you've configured this for Sidekiq as mentioned above.
## `gitlab-shell.log`
-This file lives in `/var/log/gitlab/gitaly/gitlab-shell.log` for
-Omnibus GitLab packages or in `/home/git/gitaly/gitlab-shell.log` for
-installations from source.
+GitLab Shell is used by GitLab for executing Git commands and provide SSH access to Git repositories.
+
+### For GitLab versions 12.10 and up
+
+For GitLab version 12.10 and later, there are 2 `gitlab-shell.log` files. Information containing `git-{upload-pack,receive-pack}` requests lives in `/var/log/gitlab/gitlab-shell/gitlab-shell.log`. Information about hooks to GitLab Shell from Gitaly lives in `/var/log/gitlab/gitaly/gitlab-shell.log`.
+
+Example log entries for `/var/log/gitlab/gitlab-shell/gitlab-shell.log`:
+
+```json
+{
+ "duration_ms": 74.104,
+ "level": "info",
+ "method": "POST",
+ "msg": "Finished HTTP request",
+ "time": "2020-04-17T20:28:46Z",
+ "url": "http://127.0.0.1:8080/api/v4/internal/allowed"
+}
+{
+ "command": "git-upload-pack",
+ "git_protocol": "",
+ "gl_project_path": "root/example",
+ "gl_repository": "project-1",
+ "level": "info",
+ "msg": "executing git command",
+ "time": "2020-04-17T20:28:46Z",
+ "user_id": "user-1",
+ "username": "root"
+}
+```
+
+Example log entries for `/var/log/gitlab/gitaly/gitlab-shell.log`:
+
+```json
+{
+ "method": "POST",
+ "url": "http://127.0.0.1:8080/api/v4/internal/allowed",
+ "duration": 0.058012959,
+ "gitaly_embedded": true,
+ "pid": 16636,
+ "level": "info",
+ "msg": "finished HTTP request",
+ "time": "2020-04-17T20:29:08+00:00"
+}
+{
+ "method": "POST",
+ "url": "http://127.0.0.1:8080/api/v4/internal/pre_receive",
+ "duration": 0.031022552,
+ "gitaly_embedded": true,
+ "pid": 16636,
+ "level": "info",
+ "msg": "finished HTTP request",
+ "time": "2020-04-17T20:29:08+00:00"
+}
+```
+
+### For GitLab versions 12.5 through 12.9
+
+For GitLab 12.5 to 12.9, this file lives in `/var/log/gitlab/gitaly/gitlab-shell.log` for Omnibus GitLab packages or in `/home/git/gitaly/gitlab-shell.log` for installations from source.
+
+Example log entries:
+
+```json
+{
+ "method": "POST",
+ "url": "http://127.0.0.1:8080/api/v4/internal/post_receive",
+ "duration": 0.031809164,
+ "gitaly_embedded": true,
+ "pid": 27056,
+ "level": "info",
+ "msg": "finished HTTP request",
+ "time": "2020-04-17T16:24:38+00:00"
+}
+```
+
+### For GitLab 12.5 and earlier
-NOTE: **Note:**
For GitLab 12.5 and earlier, the file lives in `/var/log/gitlab/gitlab-shell/gitlab-shell.log`.
-GitLab Shell is used by GitLab for executing Git commands and provide
-SSH access to Git repositories. For example:
+Example log entries:
```plaintext
I, [2015-02-13T06:17:00.671315 #9291] INFO -- : Adding project root/example.git at </var/opt/gitlab/git-data/repositories/root/dcdcdcdcd.git>.
@@ -447,15 +530,29 @@ I, [2015-02-13T06:17:00.679433 #9291] INFO -- : Moving existing hooks directory
User clone/fetch activity using SSH transport appears in this log as `executing git command <gitaly-upload-pack...`.
-## `current`
+## Gitaly Logs
-This file lives in `/var/log/gitlab/gitaly/current` and is produced by [runit](http://smarden.org/runit/). `runit` is packaged with Omnibus and a brief explanation of its purpose is available [in the omnibus documentation](https://docs.gitlab.com/omnibus/architecture/#runit). [Log files are rotated](http://smarden.org/runit/svlogd.8.html), renamed in unix timestamp format and `gzip`-compressed (e.g. `@1584057562.s`).
+This file lives in `/var/log/gitlab/gitaly/current` and is produced by [runit](http://smarden.org/runit/). `runit` is packaged with Omnibus and a brief explanation of its purpose is available [in the omnibus documentation](https://docs.gitlab.com/omnibus/architecture/#runit). [Log files are rotated](http://smarden.org/runit/svlogd.8.html), renamed in Unix timestamp format and `gzip`-compressed (e.g. `@1584057562.s`).
-## `unicorn_stderr.log`
+### `grpc.log`
-This file lives in `/var/log/gitlab/unicorn/unicorn_stderr.log` for
-Omnibus GitLab packages or in `/home/git/gitlab/log/unicorn_stderr.log` for
-installations from source.
+This file lives in `/var/log/gitlab/gitlab-rails/grpc.log` for Omnibus GitLab packages. Native [gRPC](https://grpc.io/) logging used by Gitaly.
+
+## `puma_stderr.log` & `puma_stdout.log`
+
+This file lives in `/var/log/gitlab/puma/puma_stderr.log` and `/var/log/gitlab/puma/puma_stdout.log` for
+Omnibus GitLab packages or in `/home/git/gitlab/log/puma_stderr.log` and `/home/git/gitlab/log/puma_stdout.log`
+for installations from source.
+
+## `unicorn_stderr.log` & `unicorn_stdout.log`
+
+NOTE: **Note:**
+Starting with GitLab 13.0, Puma is the default web server used in GitLab
+all-in-one package based installations as well as GitLab Helm chart deployments.
+
+This file lives in `/var/log/gitlab/unicorn/unicorn_stderr.log` and `/var/log/gitlab/unicorn/unicorn_stdout.log` for
+Omnibus GitLab packages or in `/home/git/gitlab/log/unicorn_stderr.log` and `/home/git/gitlab/log/unicorn_stdout.log`
+for installations from source.
Unicorn is a high-performance forking Web server which is used for
serving the GitLab application. You can look at this log if, for
@@ -484,7 +581,7 @@ This file lives in `/var/log/gitlab/gitlab-rails/repocheck.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/repocheck.log` for
installations from source.
-It logs information whenever a [repository check is run][repocheck] on a project.
+It logs information whenever a [repository check is run](repository_checks.md) on a project.
## `importer.log`
@@ -494,6 +591,8 @@ This file lives in `/var/log/gitlab/gitlab-rails/importer.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/importer.log` for
installations from source.
+It logs the progress of the import process.
+
## `auth.log`
> Introduced in GitLab 12.0.
@@ -504,9 +603,9 @@ installations from source.
This log records:
-- Information whenever [Rack Attack] registers an abusive request.
-- Requests over the [Rate Limit] on raw endpoints.
-- [Protected paths] abusive requests.
+- Information whenever [Rack Attack](../security/rack_attack.md) registers an abusive request.
+- Requests over the [Rate Limit](../user/admin_area/settings/rate_limits_on_raw_endpoints.md) on raw endpoints.
+- [Protected paths](../user/admin_area/settings/protected_paths.md) abusive requests.
NOTE: **Note:**
From [%12.1](https://gitlab.com/gitlab-org/gitlab-foss/issues/62756), user ID and username are also
@@ -523,7 +622,7 @@ installations from source.
GraphQL queries are recorded in that file. For example:
```json
-{"query_string":"query IntrospectionQuery{__schema {queryType { name },mutationType { name }}}...(etc)","variables":{"a":1,"b":2},"complexity":181,"depth":1,"duration":7}
+{"query_string":"query IntrospectionQuery{__schema {queryType { name },mutationType { name }}}...(etc)","variables":{"a":1,"b":2},"complexity":181,"depth":1,"duration_s":7}
```
## `migrations.log`
@@ -538,7 +637,7 @@ installations from source.
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19186) in GitLab 12.6.
-This file lives in `/var/log/gitlab/mail_room/mail_room_json.log` for
+This file lives in `/var/log/gitlab/mailroom/mail_room_json.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/mail_room_json.log` for
installations from source.
@@ -562,7 +661,7 @@ will be generated in `/var/log/gitlab/gitlab-rails/sidekiq_exporter.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/sidekiq_exporter.log` for
installations from source.
-If Prometheus metrics and the Web Exporter are both enabled, Unicorn/Puma will
+If Prometheus metrics and the Web Exporter are both enabled, Puma/Unicorn will
start a Web server and listen to the defined port (default: `8083`). Access logs
will be generated in `/var/log/gitlab/gitlab-rails/web_exporter.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/web_exporter.log` for
@@ -578,7 +677,7 @@ It's stored at:
- `/var/log/gitlab/gitlab-rails/database_load_balancing.log` for Omnibus GitLab packages.
- `/home/git/gitlab/log/database_load_balancing.log` for installations from source.
-## `elasticsearch.log`
+## `elasticsearch.log` **(STARTER ONLY)**
> Introduced in GitLab 12.6.
@@ -648,7 +747,24 @@ Each line contains a JSON line that can be ingested by Elasticsearch. For exampl
}
```
-## `geo.log`
+## `service_measurement.log`
+
+> Introduced in GitLab 13.0.
+
+This file lives in `/var/log/gitlab/gitlab-rails/service_measurement.log` for
+Omnibus GitLab packages or in `/home/git/gitlab/log/service_measurement.log` for
+installations from source.
+
+It contain only a single structured log with measurements for each service execution.
+It will contain measurement such as: number of sql calls, execution_time, gc_stats, memory usage, etc...
+
+For example:
+
+```json
+{ "severity":"INFO", "time":"2020-04-22T16:04:50.691Z","correlation_id":"04f1366e-57a1-45b8-88c1-b00b23dc3616","class":"Projects::ImportExport::ExportService","current_user":"John Doe","project_full_path":"group1/test-export","file_path":"/path/to/archive","gc_stats":{"count":{"before":127,"after":127,"diff":0},"heap_allocated_pages":{"before":10369,"after":10369,"diff":0},"heap_sorted_length":{"before":10369,"after":10369,"diff":0},"heap_allocatable_pages":{"before":0,"after":0,"diff":0},"heap_available_slots":{"before":4226409,"after":4226409,"diff":0},"heap_live_slots":{"before":2542709,"after":2641420,"diff":98711},"heap_free_slots":{"before":1683700,"after":1584989,"diff":-98711},"heap_final_slots":{"before":0,"after":0,"diff":0},"heap_marked_slots":{"before":2542704,"after":2542704,"diff":0},"heap_eden_pages":{"before":10369,"after":10369,"diff":0},"heap_tomb_pages":{"before":0,"after":0,"diff":0},"total_allocated_pages":{"before":10369,"after":10369,"diff":0},"total_freed_pages":{"before":0,"after":0,"diff":0},"total_allocated_objects":{"before":24896308,"after":24995019,"diff":98711},"total_freed_objects":{"before":22353599,"after":22353599,"diff":0},"malloc_increase_bytes":{"before":140032,"after":6650240,"diff":6510208},"malloc_increase_bytes_limit":{"before":25804104,"after":25804104,"diff":0},"minor_gc_count":{"before":94,"after":94,"diff":0},"major_gc_count":{"before":33,"after":33,"diff":0},"remembered_wb_unprotected_objects":{"before":34284,"after":34284,"diff":0},"remembered_wb_unprotected_objects_limit":{"before":68568,"after":68568,"diff":0},"old_objects":{"before":2404725,"after":2404725,"diff":0},"old_objects_limit":{"before":4809450,"after":4809450,"diff":0},"oldmalloc_increase_bytes":{"before":140032,"after":6650240,"diff":6510208},"oldmalloc_increase_bytes_limit":{"before":68537556,"after":68537556,"diff":0}},"time_to_finish":0.12298400001600385,"number_of_sql_calls":70,"memory_usage":"0.0 MiB","label":"process_48616"}
+```
+
+## `geo.log` **(PREMIUM ONLY)**
> Introduced in 9.5.
@@ -677,7 +793,9 @@ For Omnibus installations, NGINX logs reside in:
- `/var/log/gitlab/nginx/gitlab_pages_access.log` contains a log of requests made to Pages static sites.
- `/var/log/gitlab/nginx/gitlab_pages_error.log` contains a log of NGINX errors for Pages static sites.
- `/var/log/gitlab/nginx/gitlab_registry_access.log` contains a log of requests made to the Container Registry.
-- `/var/log/gitlab/nginx/gitlab_registry_error.log` contains a log of NGINX errors for the Container Regsitry.
+- `/var/log/gitlab/nginx/gitlab_registry_error.log` contains a log of NGINX errors for the Container Registry.
+- `/var/log/gitlab/nginx/gitlab_mattermost_access.log` contains a log of requests made to Mattermost.
+- `/var/log/gitlab/nginx/gitlab_mattermost_error.log` contains a log of NGINX errors for Mattermost.
Below is the default GitLab NGINX access log format:
@@ -685,7 +803,51 @@ Below is the default GitLab NGINX access log format:
$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"
```
-[repocheck]: repository_checks.md
-[Rack Attack]: ../security/rack_attack.md
-[Rate Limit]: ../user/admin_area/settings/rate_limits_on_raw_endpoints.md
-[Protected paths]: ../user/admin_area/settings/protected_paths.md
+## Pages Logs
+
+For Omnibus installations, Pages logs reside in `/var/log/gitlab/gitlab-pages/current`.
+
+For example:
+
+```json
+{
+ "level": "info",
+ "msg": "GitLab Pages Daemon",
+ "revision": "52b2899",
+ "time": "2020-04-22T17:53:12Z",
+ "version": "1.17.0"
+}
+{
+ "level": "info",
+ "msg": "URL: https://gitlab.com/gitlab-org/gitlab-pages",
+ "time": "2020-04-22T17:53:12Z"
+}
+{
+ "gid": 998,
+ "in-place": false,
+ "level": "info",
+ "msg": "running the daemon as unprivileged user",
+ "time": "2020-04-22T17:53:12Z",
+ "uid": 998
+}
+```
+
+## Workhorse Logs
+
+For Omnibus installations, Workhorse logs reside in `/var/log/gitlab/gitlab-workhorse/current`.
+
+## PostgreSQL Logs
+
+For Omnibus installations, PostgreSQL logs reside in `/var/log/gitlab/postgresql/current`.
+
+## Prometheus Logs
+
+For Omnibus installations, Prometheus logs reside in `/var/log/gitlab/prometheus/current`.
+
+## Redis Logs
+
+For Omnibus installations, Redis logs reside in `/var/log/gitlab/redis/current`.
+
+## Mattermost Logs
+
+For Omnibus installations, Mattermost logs reside in `/var/log/gitlab/mattermost/mattermost.log`.