diff options
Diffstat (limited to 'doc/administration/job_logs.md')
| -rw-r--r-- | doc/administration/job_logs.md | 241 |
1 files changed, 154 insertions, 87 deletions
diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index 9b25c70716b..c8702260ccb 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -7,8 +7,6 @@ type: reference # Job logs **(FREE SELF)** -> [Renamed from job traces to job logs](https://gitlab.com/gitlab-org/gitlab/-/issues/29121) in GitLab 12.5. - Job logs are sent by a runner while it's processing a job. You can see logs in job pages, pipelines, email notifications, and so on. @@ -23,82 +21,125 @@ In the following table you can see the phases a log goes through: | 2: archiving | archived log | After a job is finished | Sidekiq moves log to artifacts folder | `#{ROOT_PATH}/gitlab-rails/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` | | 3: uploading | archived log | After a log is archived | Sidekiq moves archived log to [object storage](#uploading-logs-to-object-storage) (if configured) | `#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` | -The `ROOT_PATH` varies per environment. For Omnibus GitLab it -would be `/var/opt/gitlab`, and for installations from source -it would be `/home/git/gitlab`. +The `ROOT_PATH` varies per environment: + +- For the Linux package it's `/var/opt/gitlab`. +- For self-compiled installations it's `/home/git/gitlab`. ## Changing the job logs local location -To change the location where the job logs are stored, follow the steps below. +NOTE: +For Docker installations, you can change the path where your data is mounted. +For the Helm chart, use object storage. -**In Omnibus installations:** +To change the location where the job logs are stored: -1. Edit `/etc/gitlab/gitlab.rb` and add or amend the following line: +::Tabs - ```ruby - gitlab_ci['builds_directory'] = '/mnt/to/gitlab-ci/builds' - ``` +:::TabTitle Linux package (Omnibus) -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the - changes to take effect. +1. Optional. If you have existing job logs, pause continuous integration data + processing by temporarily stopping Sidekiq: -Alternatively, if you have existing job logs you can follow -these steps to move the logs to a new location without losing any data. + ```shell + sudo gitlab-ctl stop sidekiq + ``` -1. Pause continuous integration data processing by updating this setting in `/etc/gitlab/gitlab.rb`. - Jobs in progress are not affected, based on how [data flow](#data-flow) works. +1. Set the new storage location in `/etc/gitlab/gitlab.rb`: ```ruby - sidekiq['queue_selector'] = true - sidekiq['queue_groups'] = [ - "feature_category!=continuous_integration" - ] + gitlab_ci['builds_directory'] = '/mnt/gitlab-ci/builds' ``` -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the - changes to take effect. -1. Set the new storage location in `/etc/gitlab/gitlab.rb`: +1. Save the file and reconfigure GitLab: - ```ruby - gitlab_ci['builds_directory'] = '/mnt/to/gitlab-ci/builds' + ```shell + sudo gitlab-ctl reconfigure ``` -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the - changes to take effect. 1. Use `rsync` to move job logs from the current location to the new location: ```shell - sudo rsync -avzh --remove-source-files --ignore-existing --progress /var/opt/gitlab/gitlab-ci/builds/ /mnt/to/gitlab-ci/builds` + sudo rsync -avzh --remove-source-files --ignore-existing --progress /var/opt/gitlab/gitlab-ci/builds/ /mnt/gitlab-ci/builds/ ``` Use `--ignore-existing` so you don't override new job logs with older versions of the same log. -1. Resume continuous integration data processing by editing `/etc/gitlab/gitlab.rb` and removing the `sidekiq` setting you updated earlier. -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the - changes to take effect. + +1. If you opted to pause the continuous integration data processing, you can + start Sidekiq again: + + ```shell + sudo gitlab-ctl start sidekiq + ``` + 1. Remove the old job logs storage location: ```shell - sudo rm -rf /var/opt/gitlab/gitlab-ci/builds` + sudo rm -rf /var/opt/gitlab/gitlab-ci/builds ``` -**In installations from source:** +:::TabTitle Self-compiled (source) + +1. Optional. If you have existing job logs, pause continuous integration data + processing by temporarily stopping Sidekiq: + + ```shell + # For systems running systemd + sudo systemctl stop gitlab-sidekiq + + # For systems running SysV init + sudo service gitlab stop + ``` -1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: +1. Edit `/home/git/gitlab/config/gitlab.yml` to set the new storage location: ```yaml - gitlab_ci: - # The location where build logs are stored (default: builds/). - # Relative paths are relative to Rails.root. - builds_path: path/to/builds/ + production: &base + gitlab_ci: + builds_path: /mnt/gitlab-ci/builds + ``` + +1. Save the file and restart GitLab: + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target + + # For systems running SysV init + sudo service gitlab restart + ``` + +1. Use `rsync` to move job logs from the current location to the new location: + + ```shell + sudo rsync -avzh --remove-source-files --ignore-existing --progress /home/git/gitlab/builds/ /mnt/gitlab-ci/builds/ + ``` + + Use `--ignore-existing` so you don't override new job logs with older versions of the same log. + +1. If you opted to pause the continuous integration data processing, you can + start Sidekiq again: + + ```shell + # For systems running systemd + sudo systemctl start gitlab-sidekiq + + # For systems running SysV init + sudo service gitlab start + ``` + +1. Remove the old job logs storage location: + + ```shell + sudo rm -rf /home/git/gitlab/builds ``` -1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes - to take effect. +::EndTabs ## Uploading logs to object storage Archived logs are considered as [job artifacts](job_artifacts.md). -Therefore, when you [set up the object storage integration](job_artifacts.md#object-storage-settings), +Therefore, when you [set up the object storage integration](job_artifacts.md#using-object-storage), job logs are automatically migrated to it along with the other job artifacts. See "Phase 3: uploading" in [Data flow](#data-flow) to learn about the process. @@ -118,19 +159,45 @@ There isn't a way to automatically expire old job logs, but it's safe to remove them if they're taking up too much space. If you remove the logs manually, the job output in the UI is empty. -For example, to delete all job logs older than 60 days, run the following from a shell in your GitLab instance: +For example, to delete all job logs older than 60 days, run the following +command from a shell in your GitLab instance. + +NOTE: +For the Helm chart, use the storage management tools provided with your object +storage. WARNING: -This command permanently deletes the log files and is irreversible. +The following command permanently deletes the log files and is irreversible. + +::Tabs + +:::TabTitle Linux package (Omnibus) ```shell find /var/opt/gitlab/gitlab-rails/shared/artifacts -name "job.log" -mtime +60 -delete ``` -NOTE: -After execution, broken file references can be reported when running -[`sudo gitlab-rake gitlab:artifacts:check`](raketasks/check.md#uploaded-files-integrity). -For more information, see [delete references to missing artifacts](raketasks/check.md#delete-references-to-missing-artifacts). +:::TabTitle Docker + +Assuming you mounted `/var/opt/gitlab` to `/srv/gitlab`: + +```shell +find /srv/gitlab/gitlab-rails/shared/artifacts -name "job.log" -mtime +60 -delete +``` + +:::TabTitle Self-compiled (source) + +```shell +find /home/git/gitlab/shared/artifacts -name "job.log" -mtime +60 -delete +``` + +::EndTabs + +After the logs are deleted, you can find any broken file references by running +the Rake task that checks the +[integrity of the uploaded files](raketasks/check.md#uploaded-files-integrity). +For more information, see how to +[delete references to missing artifacts](raketasks/check.md#delete-references-to-missing-artifacts). ## Incremental logging architecture @@ -140,19 +207,48 @@ For more information, see [delete references to missing artifacts](raketasks/che > - [Recommended for production use with AWS S3](https://gitlab.com/gitlab-org/gitlab/-/issues/273498) in GitLab 13.7. > - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-incremental-logging). -By default job logs are sent from the GitLab Runner in chunks and cached temporarily on disk -in `/var/opt/gitlab/gitlab-ci/builds` by Omnibus GitLab. After the job completes, -a background job archives the job log. The log is moved to `/var/opt/gitlab/gitlab-rails/shared/artifacts/` -by default, or to object storage if configured. - -In a [scaled-out architecture](reference_architectures/index.md) with Rails and Sidekiq running on more than one -server, these two locations on the file system have to be shared using NFS. +By default, job logs are sent from the GitLab Runner in chunks and cached +temporarily on disk. After the job completes, a background job archives the job +log. The log is moved to the artifacts directory by default, or to object +storage if configured. -To eliminate both file system requirements: +In a [scaled-out architecture](reference_architectures/index.md) with Rails and +Sidekiq running on more than one server, these two locations on the file system +have to be shared using NFS, which is not recommended. Instead: -1. Configure [object storage](job_artifacts.md#object-storage-settings) for storing archived job logs. +1. Configure [object storage](job_artifacts.md#using-object-storage) for storing archived job logs. 1. [Enable the incremental logging feature](#enable-or-disable-incremental-logging), which uses Redis instead of disk space for temporary caching of job logs. +### Enable or disable incremental logging + +Before you enable the feature flag: + +- Review [the limitations of incremental logging](#limitations). +- [Enable object storage](job_artifacts.md#using-object-storage). + +To enable incremental logging: + +1. Open a [Rails console](operations/rails_console.md#starting-a-rails-console-session). +1. Enable the feature flag: + + ```ruby + Feature.enable(:ci_enable_live_trace) + ``` + + Running jobs' logs continue to be written to disk, but new jobs use + incremental logging. + +To disable incremental logging: + +1. Open a [Rails console](operations/rails_console.md#starting-a-rails-console-session). +1. Disable the feature flag: + + ```ruby + Feature.disable(:ci_enable_live_trace) + ``` + + Running jobs continue to use incremental logging, but new jobs write to the disk. + ### Technical details The data flow is the same as described in the [data flow section](#data-flow) @@ -178,36 +274,7 @@ Here is the detailed data flow: ### Limitations - [Redis Cluster is not supported](https://gitlab.com/gitlab-org/gitlab/-/issues/224171). -- You must configure [object storage for CI/CD artifacts, logs, and builds](job_artifacts.md#object-storage-settings) +- You must configure [object storage for CI/CD artifacts, logs, and builds](job_artifacts.md#using-object-storage) before you enable the feature flag. After the flag is enabled, files cannot be written to disk, and there is no protection against misconfiguration. - There is [an epic tracking other potential limitations and improvements](https://gitlab.com/groups/gitlab-org/-/epics/3791). - -### Enable or disable incremental logging - -Incremental logging is under development, but [ready for production use as of GitLab 13.6](https://gitlab.com/groups/gitlab-org/-/epics/4275). It is -deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](feature_flags.md) -can enable it. - -Before you enable the feature flag: - -- Review [the limitations of incremental logging](#limitations). -- [Enable object storage](job_artifacts.md#object-storage-settings). - -To enable incremental logging: - -```ruby -Feature.enable(:ci_enable_live_trace) -``` - -Running jobs' logs continue to be written to disk, but new jobs use -incremental logging. - -To disable incremental logging: - -```ruby -Feature.disable(:ci_enable_live_trace) -``` - -Running jobs continue to use incremental logging, but new jobs write to the disk. |