From 790c68680893b333c0a9bedca7c528147b8417b1 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Mon, 18 Jan 2016 16:55:14 +0100 Subject: Added documentation for GitLab Metrics --- doc/README.md | 7 +++ doc/metrics/gitlab_configuration.md | 12 +++++ doc/metrics/influxdb_configuration.md | 96 +++++++++++++++++++++++++++++++++++ doc/metrics/influxdb_schema.md | 75 +++++++++++++++++++++++++++ doc/metrics/introduction.md | 55 ++++++++++++++++++++ 5 files changed, 245 insertions(+) create mode 100644 doc/metrics/gitlab_configuration.md create mode 100644 doc/metrics/influxdb_configuration.md create mode 100644 doc/metrics/influxdb_schema.md create mode 100644 doc/metrics/introduction.md (limited to 'doc') diff --git a/doc/README.md b/doc/README.md index 7d4f84857e0..b9ee7de5349 100644 --- a/doc/README.md +++ b/doc/README.md @@ -49,6 +49,13 @@ - [Test Clojure applications](ci/examples/test-clojure-application.md) - Help your favorite programming language and GitLab by sending a merge request with a guide for that language. +## GitLab Metrics + +- [Introduction](metrics/introduction.md) +- [GitLab Configuration](metrics/gitlab_configuration.md) +- [InfluxDB Configuration](metrics/influxdb_configuration.md) +- [InfluxDB Schema](metrics/influxdb_schema.md) + ## Administrator documentation - [Custom git hooks](hooks/custom_hooks.md) Custom git hooks (on the filesystem) for when web hooks aren't enough. diff --git a/doc/metrics/gitlab_configuration.md b/doc/metrics/gitlab_configuration.md new file mode 100644 index 00000000000..3b44b4394ff --- /dev/null +++ b/doc/metrics/gitlab_configuration.md @@ -0,0 +1,12 @@ +# GitLab Configuration + +By default GitLab Metrics is disabled. To enable GitLab Metrics and change any +of its settings open a web browser and navigate to +`http://YOUR_GITLAB_HOST/admin/application_settings`, the settings can be found +in the "Metrics" section. A restart of all GitLab processes is required for any +changes to take effect. + +## Pending Migrations + +When any migrations are pending the metrics are disabled until the migrations +have been performed. diff --git a/doc/metrics/influxdb_configuration.md b/doc/metrics/influxdb_configuration.md new file mode 100644 index 00000000000..c67c0f15108 --- /dev/null +++ b/doc/metrics/influxdb_configuration.md @@ -0,0 +1,96 @@ +# InfluxDB Configuration + +The default settings provided by InfluxDB are not sufficient for a high traffic +GitLab environment. The settings discussed in this document are based on the +settings GitLab uses for GitLab.com, depending on your own needs you may need to +further adjust them. + +## Requirements + +* InfluxDB 0.9 or newer +* A fairly modern version of Linux +* At least 4GB of RAM +* At least 10GB of storage for InfluxDB data + +Note that the RAM and storage requirements can differ greatly depending on the +amount of data received/stored. To limit the amount of stored data users can +look into [InfluxDB Retention Policies][influxdb-retention]. + +## InfluxDB Server Settings + +Since InfluxDB has many settings that users may wish to customize themselves +(e.g. what port to run InfluxDB on) we'll only cover the essentials. + +### Storage Engine + +InfluxDB comes with different storage engines and as of InfluxDB 0.9 a new +storage engine is available called "tsm1". All users _must_ use the new tsm1 +storage engine (this will be the default engine in upcoming InfluxDB engines). + +### Admin Panel + +Production environments should have the InfluxDB admin panel _disabled_. This +feature can be disabled by adding the following to your InfluxDB configuration +file: + + [admin] + enabled = false + +### HTTP + +HTTP is required when using the InfluxDB CLI or other tools such as Grafana, +thus it should be enabled. When enabling make sure to _also_ enable +authentication: + + [http] + enabled = true + auth-enabled = true + +### UDP + +GitLab writes data to InfluxDB via UDP and thus this must be enabled. Enabling +UDP can be done using the following settings: + + [udp] + enabled = true + bind-address = ":8089" + database = "gitlab" + batch-size = 1000 + batch-pending = 5 + batch-timeout = 1s + read-buffer = 209715200 + +This does the following: + +1. Enable UDP and bind it to port 8089 for all addresses. +2. Store any data received in the "gitlab" database. +3. Define a batch of points to be 1000 points in size and allow a maximum of + 5 batches _or_ flush them automatically after 1 second. +4. Define a UDP read buffer size of 200 MB. + +One of the most important settings here is the UDP read buffer size as if this +value is set too low packets will be dropped. You must also make sure the OS +buffer size is set to the same value, the default value is almost never enough. + +To set the OS buffer size to 200 MB on Linux you can run the following command: + + sysctl -w net.core.rmem_max=209715200 + +To make this permanent, add the following to `/etc/sysctl.conf` and restart the +server: + + net.core.rmem_max=209715200 + +It is **very important** to make sure the buffer sizes are large enough to +handle all data sent to InfluxDB as otherwise you _will_ lose data. The above +buffer sizes are based on the traffic for GitLab.com. Depending on the amount of +traffic users may be able to use a smaller buffer size, but we highly recommend +using _at least_ 100 MB. + +When enabling UDP users should take care to not expose the port to the public as +doing so will allow anybody to write data into your InfluxDB database (as +InfluxDB's UDP protocol doesn't support authentication). We recommend either +whitelisting the allowed IP addresses/ranges, or setting up a VLAN and only +allowing traffic from members of said VLAN. + +[influxdb-retention]: https://docs.influxdata.com/influxdb/v0.9/query_language/database_management/#retention-policy-management diff --git a/doc/metrics/influxdb_schema.md b/doc/metrics/influxdb_schema.md new file mode 100644 index 00000000000..a8e69b5e173 --- /dev/null +++ b/doc/metrics/influxdb_schema.md @@ -0,0 +1,75 @@ +# InfluxDB Schema + +The following measurements are currently stored in InfluxDB: + +* `PROCESS_file_descriptors` +* `PROCESS_gc_statistics` +* `PROCESS_memory_usage` +* `PROCESS_method_calls` +* `PROCESS_object_counts` +* `PROCESS_transactions` +* `PROCESS_views` + +Here `PROCESS` is replaced with either "rails" or "sidekiq" depending on the +process type. In all series any form of duration is stored in milliseconds. + +## PROCESS_file_descriptors + +This measurement contains the number of open file descriptors over time. The +value field `value` contains the number of descriptors. + +## PROCESS_gc_statistics + +This measurement contains Ruby garbage collection statistics such as the amount +of minor/major GC runs (relative to the last sampling interval), the time spent +in garbage collection cycles, and all fields/values returned by `GC.stat`. + +## PROCESS_memory_usage + +This measurement contains the process' memory usage (in bytes) over time. The +value field `value` contains the number of bytes. + +## PROCESS_method_calls + +This measurement contains the methods called during a transaction along with +their durations and a name of the transaction action that invoked the method (if +available). The method call duration is stored in the value field `duration` +while the method name is stored in the tag `method`. The tag `action` contains +the full name of the transaction action. Both the `method` and `action` fields +are in the following format: + + ClassName#method_name + +For example, a method called by the `show` method in the `UsersController` class +would have `action` set to `UsersController#show`. + +## PROCESS_object_counts + +This measurement is used to store retained Ruby objects (per class) and the +amount of retained objects. The number of objects is stored in the `count` value +field while the class name is stored in the `type` tag. + +## PROCESS_transactions + +This measurement is used to store basic transaction details such as the time it +took to complete a transaction, how much time was spent in SQL queries, etc. The +following value fields are available: + +* `duration`: the total duration of the transaction. +* `allocated_memory`: the amount of bytes allocated while the transaction was + running. This value is only reliable when using single-threaded application + servers. +* `method_duration`: the total time spent in method calls. +* `sql_duration`: the total time spent in SQL queries. +* `view_duration`: the total time spent in views. + +## PROCESS_views + +This measurement is used to store view rendering timings for a transaction. The +following value fields are available: + +* `duration`: the rendering time of the view. +* `view`: the path of the view, relative to the application's root directory. + +The `action` tag contains the action name of the transaction that rendered the +view. diff --git a/doc/metrics/introduction.md b/doc/metrics/introduction.md new file mode 100644 index 00000000000..007fff9e7b4 --- /dev/null +++ b/doc/metrics/introduction.md @@ -0,0 +1,55 @@ +# Introduction to GitLab Metrics + +GitLab comes with its own application performance measuring system as of GitLab +8.4, simply called "GitLab Metrics". GitLab Metrics is available in both the +Community and Enterprise editions. + +GitLab Metrics makes it possible to measure a wide variety of statistics +including (but not limited to): + +* The time it took to complete a transaction (a web request or Sidekiq job). +* The time spent in running SQL queries and rendering HAML views. +* The time spent executing (instrumented) Ruby methods. +* Ruby object allocations, and retained objects in particular. +* System statistics such as the process' memory usage and open file descriptors. +* Ruby garbage collection statistics. + +Metrics data is written to [InfluxDB][influxdb] over [UDP](influxdb-udp). Stored +data can be visualized using [Grafana][grafana] or any other application that +supports reading data from InfluxDB. Alternatively data can be queried using the +InfluxDB CLI. + +## Metric Types + +Two types of metrics are collected: + +1. Transaction specific metrics. +2. Sampled metrics, collected at a certain interval in a separate thread. + +### Transaction Metrics + +Transaction metrics are metrics that can be associated with a single +transaction. This includes statistics such as the transaction duration, timings +of any executed SQL queries, time spent rendering HAML views, etc. These metrics +are collected for every Rack request and Sidekiq job processed. + +### Sampled Metrics + +Sampled metrics are metrics that can't be associated with a single transaction. +Examples include garbage collection statistics and retained Ruby objects. These +metrics are collected at a regular interval. This interval is made up out of two +parts: + +1. A user defined interval. +2. A randomly generated offset added on top of the interval, the same offset + can't be used twice in a row. + +The actual interval can be anywhere between a half of the defined interval and a +half above the interval. For example, for a user defined interval of 15 seconds +the actual interval can be anywhere between 7.5 and 22.5. The interval is +re-generated for every sampling run instead of being generated once and re-used +for the duration of the process' lifetime. + +[influxdb]: https://influxdata.com/time-series-platform/influxdb/ +[influxdb-udp]: https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/ +[grafana]: http://grafana.org/ -- cgit v1.2.1 From d4c05766c7a058cbf62186723216d754eebad92e Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Tue, 19 Jan 2016 22:18:51 +0100 Subject: Refactor GitLab Metrics docs [ci skip] --- doc/README.md | 8 +- doc/integration/README.md | 1 + doc/integration/metrics/gitlab_configuration.md | 39 +++++ .../img/metrics_gitlab_configuration_settings.png | Bin 0 -> 45148 bytes doc/integration/metrics/influxdb_configuration.md | 192 +++++++++++++++++++++ doc/integration/metrics/influxdb_schema.md | 87 ++++++++++ doc/integration/metrics/introduction.md | 64 +++++++ doc/metrics/gitlab_configuration.md | 12 -- doc/metrics/influxdb_configuration.md | 96 ----------- doc/metrics/influxdb_schema.md | 75 -------- doc/metrics/introduction.md | 55 ------ 11 files changed, 384 insertions(+), 245 deletions(-) create mode 100644 doc/integration/metrics/gitlab_configuration.md create mode 100644 doc/integration/metrics/img/metrics_gitlab_configuration_settings.png create mode 100644 doc/integration/metrics/influxdb_configuration.md create mode 100644 doc/integration/metrics/influxdb_schema.md create mode 100644 doc/integration/metrics/introduction.md delete mode 100644 doc/metrics/gitlab_configuration.md delete mode 100644 doc/metrics/influxdb_configuration.md delete mode 100644 doc/metrics/influxdb_schema.md delete mode 100644 doc/metrics/introduction.md (limited to 'doc') diff --git a/doc/README.md b/doc/README.md index b9ee7de5349..bee9ab05906 100644 --- a/doc/README.md +++ b/doc/README.md @@ -49,13 +49,6 @@ - [Test Clojure applications](ci/examples/test-clojure-application.md) - Help your favorite programming language and GitLab by sending a merge request with a guide for that language. -## GitLab Metrics - -- [Introduction](metrics/introduction.md) -- [GitLab Configuration](metrics/gitlab_configuration.md) -- [InfluxDB Configuration](metrics/influxdb_configuration.md) -- [InfluxDB Schema](metrics/influxdb_schema.md) - ## Administrator documentation - [Custom git hooks](hooks/custom_hooks.md) Custom git hooks (on the filesystem) for when web hooks aren't enough. @@ -74,6 +67,7 @@ - [Reply by email](incoming_email/README.md) Allow users to comment on issues and merge requests by replying to notification emails. - [Migrate GitLab CI to CE/EE](migrate_ci_to_ce/README.md) Follow this guide to migrate your existing GitLab CI data to GitLab CE/EE. - [Git LFS configuration](workflow/lfs/lfs_administration.md) +- [GitLab Metrics](integration/metrics/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics ## Contributor documentation diff --git a/doc/integration/README.md b/doc/integration/README.md index 5edac746c7b..5ba0d0fb245 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -15,6 +15,7 @@ See the documentation below for details on how to configure these services. - [OAuth2 provider](oauth_provider.md) OAuth2 application creation - [Gmail actions buttons](gmail_action_buttons_for_gitlab.md) Adds GitLab actions to messages - [reCAPTCHA](recaptcha.md) Configure GitLab to use Google reCAPTCHA for new users +- [GitLab Metrics](metrics/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics GitLab Enterprise Edition contains [advanced Jenkins support][jenkins]. diff --git a/doc/integration/metrics/gitlab_configuration.md b/doc/integration/metrics/gitlab_configuration.md new file mode 100644 index 00000000000..019db1ad02c --- /dev/null +++ b/doc/integration/metrics/gitlab_configuration.md @@ -0,0 +1,39 @@ +# GitLab Configuration + +GitLab Metrics is disabled by default. To enable it and change any of its +settings, navigate to the Admin area in **Settings > Metrics** +(`/admin/application_settings`). + +The minimum required settings you need to set are the InfluxDB host and port. +Make sure _Enable InfluxDB Metrics_ is checked and hit **Save** to save the +changes. + +--- + +![GitLab Metrics Admin Settings](img/metrics_gitlab_configuration_settings.png) + +--- + +Finally, a restart of all GitLab processes is required for the changes to take +effect: + +```bash +# For Omnibus installations +sudo gitlab-ctl restart + +# For installations from source +sudo service gitlab restart +``` + +## Pending Migrations + +When any migrations are pending, the metrics are disabled until the migrations +have been performed. + +--- + +Read more on: + +- [Introduction to GitLab Metrics](introduction.md) +- [InfluxDB Configuration](influxdb_configuration.md) +- [InfluxDB Schema](influxdb_schema.md) diff --git a/doc/integration/metrics/img/metrics_gitlab_configuration_settings.png b/doc/integration/metrics/img/metrics_gitlab_configuration_settings.png new file mode 100644 index 00000000000..14d82b6ac98 Binary files /dev/null and b/doc/integration/metrics/img/metrics_gitlab_configuration_settings.png differ diff --git a/doc/integration/metrics/influxdb_configuration.md b/doc/integration/metrics/influxdb_configuration.md new file mode 100644 index 00000000000..13227f1e889 --- /dev/null +++ b/doc/integration/metrics/influxdb_configuration.md @@ -0,0 +1,192 @@ +# InfluxDB Configuration + +The default settings provided by [InfluxDB] are not sufficient for a high traffic +GitLab environment. The settings discussed in this document are based on the +settings GitLab uses for GitLab.com, depending on your own needs you may need to +further adjust them. + +If you are intending to run InfluxDB on the same server as GitLab, make sure +you have plenty of RAM since InfluxDB can use quite a bit depending on traffic. + +Unless you are going with a budget setup, it's advised to run it separately. + +## Requirements + +- InfluxDB 0.9.5 or newer +- A fairly modern version of Linux +- At least 4GB of RAM +- At least 10GB of storage for InfluxDB data + +Note that the RAM and storage requirements can differ greatly depending on the +amount of data received/stored. To limit the amount of stored data users can +look into [InfluxDB Retention Policies][influxdb-retention]. + +## Installation + +Installing InfluxDB is out of the scope of this document. Please refer to the +[InfluxDB documentation]. + +## InfluxDB Server Settings + +Since InfluxDB has many settings that users may wish to customize themselves +(e.g. what port to run InfluxDB on), we'll only cover the essentials. + +The configuration file in question is usually located at +`/etc/influxdb/influxdb.conf`. Whenever you make a change in this file, +InfluxDB needs to be restarted. + +### Storage Engine + +InfluxDB comes with different storage engines and as of InfluxDB 0.9.5 a new +storage engine is available, called [TSM Tree]. All users **must** use the new +`tsm1` storage engine as this [will be the default engine][tsm1-commit] in +upcoming InfluxDB releases. + +Make sure you have the following in your configuration file: + +``` +[data] + dir = "/var/lib/influxdb/data" + engine = "tsm1" +``` + +### Admin Panel + +Production environments should have the InfluxDB admin panel **disabled**. This +feature can be disabled by adding the following to your InfluxDB configuration +file: + +``` +[admin] + enabled = false +``` + +### HTTP + +HTTP is required when using the [InfluxDB CLI] or other tools such as Grafana, +thus it should be enabled. When enabling make sure to _also_ enable +authentication: + +``` +[http] + enabled = true + auth-enabled = true +``` + +_**Note:** Before you enable authentication, you might want to [create an +admin user](#create-a-new-admin-user)._ + +### UDP + +GitLab writes data to InfluxDB via UDP and thus this must be enabled. Enabling +UDP can be done using the following settings: + +``` +[[udp]] + enabled = true + bind-address = ":8089" + database = "gitlab" + batch-size = 1000 + batch-pending = 5 + batch-timeout = "1s" + read-buffer = 209715200 +``` + +This does the following: + +1. Enable UDP and bind it to port 8089 for all addresses. +2. Store any data received in the "gitlab" database. +3. Define a batch of points to be 1000 points in size and allow a maximum of + 5 batches _or_ flush them automatically after 1 second. +4. Define a UDP read buffer size of 200 MB. + +One of the most important settings here is the UDP read buffer size as if this +value is set too low, packets will be dropped. You must also make sure the OS +buffer size is set to the same value, the default value is almost never enough. + +To set the OS buffer size to 200 MB, on Linux you can run the following command: + +```bash +sysctl -w net.core.rmem_max=209715200 +``` + +To make this permanent, add the following to `/etc/sysctl.conf` and restart the +server: + +```bash +net.core.rmem_max=209715200 +``` + +It is **very important** to make sure the buffer sizes are large enough to +handle all data sent to InfluxDB as otherwise you _will_ lose data. The above +buffer sizes are based on the traffic for GitLab.com. Depending on the amount of +traffic, users may be able to use a smaller buffer size, but we highly recommend +using _at least_ 100 MB. + +When enabling UDP, users should take care to not expose the port to the public, +as doing so will allow anybody to write data into your InfluxDB database (as +[InfluxDB's UDP protocol][udp] doesn't support authentication). We recommend either +whitelisting the allowed IP addresses/ranges, or setting up a VLAN and only +allowing traffic from members of said VLAN. + +## Create a new admin user + +If you want to [enable authentication](#http), you might want to [create an +admin user][influx-admin]: + +``` +influx -execute "CREATE USER thedude WITH PASSWORD '1234' WITH ALL PRIVILEGES" +``` + +## Create the `gitlab` database + +Once you get InfluxDB up and running, you need to create a database for GitLab. +Make sure you have changed the [storage engine](#storage-engine) to `tsm1` +before creating a database. + +_**Note:** If you [created an admin user](#create-a-new-admin-user) and enabled +[HTTP authentication](#http), remember to append the username (`-username thedude`) +and password (`-password 1234`) to the commands below._ + +Run the following command to create a database named `gitlab`: + +```bash +influx -execute 'CREATE DATABASE gitlab' +``` + +The name **must** be `gitlab`, do not use any other name. + +Next, make sure that the database was successfully created: + +```bash +influx -execute 'SHOW DATABASES' +``` + +The output should be similar to: + +``` +name: databases +--------------- +name +_internal +gitlab +``` + +That's it! Now your GitLab instance should send data to InfluxDB. + +--- + +Read more on: + +- [Introduction to GitLab Metrics](introduction.md) +- [GitLab Configuration](gitlab_configuration.md) +- [InfluxDB Schema](influxdb_schema.md) + +[influxdb-retention]: https://docs.influxdata.com/influxdb/v0.9/query_language/database_management/#retention-policy-management +[influxdb documentation]: https://docs.influxdata.com/influxdb/v0.9/ +[influxdb cli]: https://docs.influxdata.com/influxdb/v0.9/tools/shell/ +[udp]: https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/ +[influxdb]: https://influxdata.com/time-series-platform/influxdb/ +[tsm tree]: https://influxdata.com/blog/new-storage-engine-time-structured-merge-tree/ +[tsm1-commit]: https://github.com/influxdata/influxdb/commit/15d723dc77651bac83e09e2b1c94be480966cb0d +[influx-admin]: https://docs.influxdata.com/influxdb/v0.9/administration/authentication_and_authorization/#create-a-new-admin-user diff --git a/doc/integration/metrics/influxdb_schema.md b/doc/integration/metrics/influxdb_schema.md new file mode 100644 index 00000000000..a9ef0b446c8 --- /dev/null +++ b/doc/integration/metrics/influxdb_schema.md @@ -0,0 +1,87 @@ +# InfluxDB Schema + +The following measurements are currently stored in InfluxDB: + +- `PROCESS_file_descriptors` +- `PROCESS_gc_statistics` +- `PROCESS_memory_usage` +- `PROCESS_method_calls` +- `PROCESS_object_counts` +- `PROCESS_transactions` +- `PROCESS_views` + +Here, `PROCESS` is replaced with either `rails` or `sidekiq` depending on the +process type. In all series, any form of duration is stored in milliseconds. + +## PROCESS_file_descriptors + +This measurement contains the number of open file descriptors over time. The +value field `value` contains the number of descriptors. + +## PROCESS_gc_statistics + +This measurement contains Ruby garbage collection statistics such as the amount +of minor/major GC runs (relative to the last sampling interval), the time spent +in garbage collection cycles, and all fields/values returned by `GC.stat`. + +## PROCESS_memory_usage + +This measurement contains the process' memory usage (in bytes) over time. The +value field `value` contains the number of bytes. + +## PROCESS_method_calls + +This measurement contains the methods called during a transaction along with +their duration, and a name of the transaction action that invoked the method (if +available). The method call duration is stored in the value field `duration`, +while the method name is stored in the tag `method`. The tag `action` contains +the full name of the transaction action. Both the `method` and `action` fields +are in the following format: + +``` +ClassName#method_name +``` + +For example, a method called by the `show` method in the `UsersController` class +would have `action` set to `UsersController#show`. + +## PROCESS_object_counts + +This measurement is used to store retained Ruby objects (per class) and the +amount of retained objects. The number of objects is stored in the `count` value +field while the class name is stored in the `type` tag. + +## PROCESS_transactions + +This measurement is used to store basic transaction details such as the time it +took to complete a transaction, how much time was spent in SQL queries, etc. The +following value fields are available: + +| Value | Description | +| ----- | ----------- | +| `duration` | The total duration of the transaction | +| `allocated_memory` | The amount of bytes allocated while the transaction was running. This value is only reliable when using single-threaded application servers | +| `method_duration` | The total time spent in method calls | +| `sql_duration` | The total time spent in SQL queries | +| `view_duration` | The total time spent in views | + +## PROCESS_views + +This measurement is used to store view rendering timings for a transaction. The +following value fields are available: + +| Value | Description | +| ----- | ----------- | +| `duration` | The rendering time of the view | +| `view` | The path of the view, relative to the application's root directory | + +The `action` tag contains the action name of the transaction that rendered the +view. + +--- + +Read more on: + +- [Introduction to GitLab Metrics](introduction.md) +- [GitLab Configuration](gitlab_configuration.md) +- [InfluxDB Configuration](influxdb_configuration.md) diff --git a/doc/integration/metrics/introduction.md b/doc/integration/metrics/introduction.md new file mode 100644 index 00000000000..bf388aa2bba --- /dev/null +++ b/doc/integration/metrics/introduction.md @@ -0,0 +1,64 @@ +# GitLab Metrics + +GitLab comes with its own application performance measuring system as of GitLab +8.4, simply called "GitLab Metrics". GitLab Metrics is available in both the +Community and Enterprise editions. + +Apart from this introduction, you are advised to read through the following +documents in order to understand and properly configure GitLab Metrics: + +- [GitLab Configuration](gitlab_configuration.md) +- [InfluxDB Configuration](influxdb_configuration.md) +- [InfluxDB Schema](influxdb_schema.md) + +## Introduction to GitLab Metrics + +GitLab Metrics makes it possible to measure a wide variety of statistics +including (but not limited to): + +- The time it took to complete a transaction (a web request or Sidekiq job). +- The time spent in running SQL queries and rendering HAML views. +- The time spent executing (instrumented) Ruby methods. +- Ruby object allocations, and retained objects in particular. +- System statistics such as the process' memory usage and open file descriptors. +- Ruby garbage collection statistics. + +Metrics data is written to [InfluxDB][influxdb] over [UDP][influxdb-udp]. Stored +data can be visualized using [Grafana][grafana] or any other application that +supports reading data from InfluxDB. Alternatively data can be queried using the +InfluxDB CLI. + +## Metric Types + +Two types of metrics are collected: + +1. Transaction specific metrics. +1. Sampled metrics, collected at a certain interval in a separate thread. + +### Transaction Metrics + +Transaction metrics are metrics that can be associated with a single +transaction. This includes statistics such as the transaction duration, timings +of any executed SQL queries, time spent rendering HAML views, etc. These metrics +are collected for every Rack request and Sidekiq job processed. + +### Sampled Metrics + +Sampled metrics are metrics that can't be associated with a single transaction. +Examples include garbage collection statistics and retained Ruby objects. These +metrics are collected at a regular interval. This interval is made up out of two +parts: + +1. A user defined interval. +1. A randomly generated offset added on top of the interval, the same offset + can't be used twice in a row. + +The actual interval can be anywhere between a half of the defined interval and a +half above the interval. For example, for a user defined interval of 15 seconds +the actual interval can be anywhere between 7.5 and 22.5. The interval is +re-generated for every sampling run instead of being generated once and re-used +for the duration of the process' lifetime. + +[influxdb]: https://influxdata.com/time-series-platform/influxdb/ +[influxdb-udp]: https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/ +[grafana]: http://grafana.org/ diff --git a/doc/metrics/gitlab_configuration.md b/doc/metrics/gitlab_configuration.md deleted file mode 100644 index 3b44b4394ff..00000000000 --- a/doc/metrics/gitlab_configuration.md +++ /dev/null @@ -1,12 +0,0 @@ -# GitLab Configuration - -By default GitLab Metrics is disabled. To enable GitLab Metrics and change any -of its settings open a web browser and navigate to -`http://YOUR_GITLAB_HOST/admin/application_settings`, the settings can be found -in the "Metrics" section. A restart of all GitLab processes is required for any -changes to take effect. - -## Pending Migrations - -When any migrations are pending the metrics are disabled until the migrations -have been performed. diff --git a/doc/metrics/influxdb_configuration.md b/doc/metrics/influxdb_configuration.md deleted file mode 100644 index c67c0f15108..00000000000 --- a/doc/metrics/influxdb_configuration.md +++ /dev/null @@ -1,96 +0,0 @@ -# InfluxDB Configuration - -The default settings provided by InfluxDB are not sufficient for a high traffic -GitLab environment. The settings discussed in this document are based on the -settings GitLab uses for GitLab.com, depending on your own needs you may need to -further adjust them. - -## Requirements - -* InfluxDB 0.9 or newer -* A fairly modern version of Linux -* At least 4GB of RAM -* At least 10GB of storage for InfluxDB data - -Note that the RAM and storage requirements can differ greatly depending on the -amount of data received/stored. To limit the amount of stored data users can -look into [InfluxDB Retention Policies][influxdb-retention]. - -## InfluxDB Server Settings - -Since InfluxDB has many settings that users may wish to customize themselves -(e.g. what port to run InfluxDB on) we'll only cover the essentials. - -### Storage Engine - -InfluxDB comes with different storage engines and as of InfluxDB 0.9 a new -storage engine is available called "tsm1". All users _must_ use the new tsm1 -storage engine (this will be the default engine in upcoming InfluxDB engines). - -### Admin Panel - -Production environments should have the InfluxDB admin panel _disabled_. This -feature can be disabled by adding the following to your InfluxDB configuration -file: - - [admin] - enabled = false - -### HTTP - -HTTP is required when using the InfluxDB CLI or other tools such as Grafana, -thus it should be enabled. When enabling make sure to _also_ enable -authentication: - - [http] - enabled = true - auth-enabled = true - -### UDP - -GitLab writes data to InfluxDB via UDP and thus this must be enabled. Enabling -UDP can be done using the following settings: - - [udp] - enabled = true - bind-address = ":8089" - database = "gitlab" - batch-size = 1000 - batch-pending = 5 - batch-timeout = 1s - read-buffer = 209715200 - -This does the following: - -1. Enable UDP and bind it to port 8089 for all addresses. -2. Store any data received in the "gitlab" database. -3. Define a batch of points to be 1000 points in size and allow a maximum of - 5 batches _or_ flush them automatically after 1 second. -4. Define a UDP read buffer size of 200 MB. - -One of the most important settings here is the UDP read buffer size as if this -value is set too low packets will be dropped. You must also make sure the OS -buffer size is set to the same value, the default value is almost never enough. - -To set the OS buffer size to 200 MB on Linux you can run the following command: - - sysctl -w net.core.rmem_max=209715200 - -To make this permanent, add the following to `/etc/sysctl.conf` and restart the -server: - - net.core.rmem_max=209715200 - -It is **very important** to make sure the buffer sizes are large enough to -handle all data sent to InfluxDB as otherwise you _will_ lose data. The above -buffer sizes are based on the traffic for GitLab.com. Depending on the amount of -traffic users may be able to use a smaller buffer size, but we highly recommend -using _at least_ 100 MB. - -When enabling UDP users should take care to not expose the port to the public as -doing so will allow anybody to write data into your InfluxDB database (as -InfluxDB's UDP protocol doesn't support authentication). We recommend either -whitelisting the allowed IP addresses/ranges, or setting up a VLAN and only -allowing traffic from members of said VLAN. - -[influxdb-retention]: https://docs.influxdata.com/influxdb/v0.9/query_language/database_management/#retention-policy-management diff --git a/doc/metrics/influxdb_schema.md b/doc/metrics/influxdb_schema.md deleted file mode 100644 index a8e69b5e173..00000000000 --- a/doc/metrics/influxdb_schema.md +++ /dev/null @@ -1,75 +0,0 @@ -# InfluxDB Schema - -The following measurements are currently stored in InfluxDB: - -* `PROCESS_file_descriptors` -* `PROCESS_gc_statistics` -* `PROCESS_memory_usage` -* `PROCESS_method_calls` -* `PROCESS_object_counts` -* `PROCESS_transactions` -* `PROCESS_views` - -Here `PROCESS` is replaced with either "rails" or "sidekiq" depending on the -process type. In all series any form of duration is stored in milliseconds. - -## PROCESS_file_descriptors - -This measurement contains the number of open file descriptors over time. The -value field `value` contains the number of descriptors. - -## PROCESS_gc_statistics - -This measurement contains Ruby garbage collection statistics such as the amount -of minor/major GC runs (relative to the last sampling interval), the time spent -in garbage collection cycles, and all fields/values returned by `GC.stat`. - -## PROCESS_memory_usage - -This measurement contains the process' memory usage (in bytes) over time. The -value field `value` contains the number of bytes. - -## PROCESS_method_calls - -This measurement contains the methods called during a transaction along with -their durations and a name of the transaction action that invoked the method (if -available). The method call duration is stored in the value field `duration` -while the method name is stored in the tag `method`. The tag `action` contains -the full name of the transaction action. Both the `method` and `action` fields -are in the following format: - - ClassName#method_name - -For example, a method called by the `show` method in the `UsersController` class -would have `action` set to `UsersController#show`. - -## PROCESS_object_counts - -This measurement is used to store retained Ruby objects (per class) and the -amount of retained objects. The number of objects is stored in the `count` value -field while the class name is stored in the `type` tag. - -## PROCESS_transactions - -This measurement is used to store basic transaction details such as the time it -took to complete a transaction, how much time was spent in SQL queries, etc. The -following value fields are available: - -* `duration`: the total duration of the transaction. -* `allocated_memory`: the amount of bytes allocated while the transaction was - running. This value is only reliable when using single-threaded application - servers. -* `method_duration`: the total time spent in method calls. -* `sql_duration`: the total time spent in SQL queries. -* `view_duration`: the total time spent in views. - -## PROCESS_views - -This measurement is used to store view rendering timings for a transaction. The -following value fields are available: - -* `duration`: the rendering time of the view. -* `view`: the path of the view, relative to the application's root directory. - -The `action` tag contains the action name of the transaction that rendered the -view. diff --git a/doc/metrics/introduction.md b/doc/metrics/introduction.md deleted file mode 100644 index 007fff9e7b4..00000000000 --- a/doc/metrics/introduction.md +++ /dev/null @@ -1,55 +0,0 @@ -# Introduction to GitLab Metrics - -GitLab comes with its own application performance measuring system as of GitLab -8.4, simply called "GitLab Metrics". GitLab Metrics is available in both the -Community and Enterprise editions. - -GitLab Metrics makes it possible to measure a wide variety of statistics -including (but not limited to): - -* The time it took to complete a transaction (a web request or Sidekiq job). -* The time spent in running SQL queries and rendering HAML views. -* The time spent executing (instrumented) Ruby methods. -* Ruby object allocations, and retained objects in particular. -* System statistics such as the process' memory usage and open file descriptors. -* Ruby garbage collection statistics. - -Metrics data is written to [InfluxDB][influxdb] over [UDP](influxdb-udp). Stored -data can be visualized using [Grafana][grafana] or any other application that -supports reading data from InfluxDB. Alternatively data can be queried using the -InfluxDB CLI. - -## Metric Types - -Two types of metrics are collected: - -1. Transaction specific metrics. -2. Sampled metrics, collected at a certain interval in a separate thread. - -### Transaction Metrics - -Transaction metrics are metrics that can be associated with a single -transaction. This includes statistics such as the transaction duration, timings -of any executed SQL queries, time spent rendering HAML views, etc. These metrics -are collected for every Rack request and Sidekiq job processed. - -### Sampled Metrics - -Sampled metrics are metrics that can't be associated with a single transaction. -Examples include garbage collection statistics and retained Ruby objects. These -metrics are collected at a regular interval. This interval is made up out of two -parts: - -1. A user defined interval. -2. A randomly generated offset added on top of the interval, the same offset - can't be used twice in a row. - -The actual interval can be anywhere between a half of the defined interval and a -half above the interval. For example, for a user defined interval of 15 seconds -the actual interval can be anywhere between 7.5 and 22.5. The interval is -re-generated for every sampling run instead of being generated once and re-used -for the duration of the process' lifetime. - -[influxdb]: https://influxdata.com/time-series-platform/influxdb/ -[influxdb-udp]: https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/ -[grafana]: http://grafana.org/ -- cgit v1.2.1 From 6f32459f2c0ebc0aa10c728e5562375db9cf4c52 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Wed, 20 Jan 2016 12:05:03 +0100 Subject: Renamed "Metrics" to "Performance Monitoring" --- doc/README.md | 2 +- doc/integration/README.md | 2 +- doc/integration/metrics/gitlab_configuration.md | 6 +++--- doc/integration/metrics/influxdb_configuration.md | 2 +- doc/integration/metrics/influxdb_schema.md | 2 +- doc/integration/metrics/introduction.md | 10 +++++----- 6 files changed, 12 insertions(+), 12 deletions(-) (limited to 'doc') diff --git a/doc/README.md b/doc/README.md index bee9ab05906..05c0d6fa0fe 100644 --- a/doc/README.md +++ b/doc/README.md @@ -67,7 +67,7 @@ - [Reply by email](incoming_email/README.md) Allow users to comment on issues and merge requests by replying to notification emails. - [Migrate GitLab CI to CE/EE](migrate_ci_to_ce/README.md) Follow this guide to migrate your existing GitLab CI data to GitLab CE/EE. - [Git LFS configuration](workflow/lfs/lfs_administration.md) -- [GitLab Metrics](integration/metrics/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics +- [GitLab Performance Monitoring](integration/metrics/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics ## Contributor documentation diff --git a/doc/integration/README.md b/doc/integration/README.md index 5ba0d0fb245..297e4f6e7bf 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -15,7 +15,7 @@ See the documentation below for details on how to configure these services. - [OAuth2 provider](oauth_provider.md) OAuth2 application creation - [Gmail actions buttons](gmail_action_buttons_for_gitlab.md) Adds GitLab actions to messages - [reCAPTCHA](recaptcha.md) Configure GitLab to use Google reCAPTCHA for new users -- [GitLab Metrics](metrics/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics +- [GitLab Performance Monitoring](metrics/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics GitLab Enterprise Edition contains [advanced Jenkins support][jenkins]. diff --git a/doc/integration/metrics/gitlab_configuration.md b/doc/integration/metrics/gitlab_configuration.md index 019db1ad02c..b856e7935a3 100644 --- a/doc/integration/metrics/gitlab_configuration.md +++ b/doc/integration/metrics/gitlab_configuration.md @@ -1,6 +1,6 @@ # GitLab Configuration -GitLab Metrics is disabled by default. To enable it and change any of its +GitLab Performance Monitoring is disabled by default. To enable it and change any of its settings, navigate to the Admin area in **Settings > Metrics** (`/admin/application_settings`). @@ -10,7 +10,7 @@ changes. --- -![GitLab Metrics Admin Settings](img/metrics_gitlab_configuration_settings.png) +![GitLab Performance Monitoring Admin Settings](img/metrics_gitlab_configuration_settings.png) --- @@ -34,6 +34,6 @@ have been performed. Read more on: -- [Introduction to GitLab Metrics](introduction.md) +- [Introduction to GitLab Performance Monitoring](introduction.md) - [InfluxDB Configuration](influxdb_configuration.md) - [InfluxDB Schema](influxdb_schema.md) diff --git a/doc/integration/metrics/influxdb_configuration.md b/doc/integration/metrics/influxdb_configuration.md index 13227f1e889..df5e4af7fad 100644 --- a/doc/integration/metrics/influxdb_configuration.md +++ b/doc/integration/metrics/influxdb_configuration.md @@ -178,7 +178,7 @@ That's it! Now your GitLab instance should send data to InfluxDB. Read more on: -- [Introduction to GitLab Metrics](introduction.md) +- [Introduction to GitLab Performance Monitoring](introduction.md) - [GitLab Configuration](gitlab_configuration.md) - [InfluxDB Schema](influxdb_schema.md) diff --git a/doc/integration/metrics/influxdb_schema.md b/doc/integration/metrics/influxdb_schema.md index a9ef0b446c8..a5a8aebd2d1 100644 --- a/doc/integration/metrics/influxdb_schema.md +++ b/doc/integration/metrics/influxdb_schema.md @@ -82,6 +82,6 @@ view. Read more on: -- [Introduction to GitLab Metrics](introduction.md) +- [Introduction to GitLab Performance Monitoring](introduction.md) - [GitLab Configuration](gitlab_configuration.md) - [InfluxDB Configuration](influxdb_configuration.md) diff --git a/doc/integration/metrics/introduction.md b/doc/integration/metrics/introduction.md index bf388aa2bba..f2460d31302 100644 --- a/doc/integration/metrics/introduction.md +++ b/doc/integration/metrics/introduction.md @@ -1,19 +1,19 @@ -# GitLab Metrics +# GitLab Performance Monitoring GitLab comes with its own application performance measuring system as of GitLab -8.4, simply called "GitLab Metrics". GitLab Metrics is available in both the +8.4, simply called "GitLab Performance Monitoring". GitLab Performance Monitoring is available in both the Community and Enterprise editions. Apart from this introduction, you are advised to read through the following -documents in order to understand and properly configure GitLab Metrics: +documents in order to understand and properly configure GitLab Performance Monitoring: - [GitLab Configuration](gitlab_configuration.md) - [InfluxDB Configuration](influxdb_configuration.md) - [InfluxDB Schema](influxdb_schema.md) -## Introduction to GitLab Metrics +## Introduction to GitLab Performance Monitoring -GitLab Metrics makes it possible to measure a wide variety of statistics +GitLab Performance Monitoring makes it possible to measure a wide variety of statistics including (but not limited to): - The time it took to complete a transaction (a web request or Sidekiq job). -- cgit v1.2.1 From 82bc7679e782059568402a282905d3d607dbb97d Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 20 Jan 2016 23:02:24 +0100 Subject: Move integration/metrics to monitoring/performance [ci skip] --- doc/README.md | 2 +- doc/integration/metrics/gitlab_configuration.md | 39 ----- .../img/metrics_gitlab_configuration_settings.png | Bin 45148 -> 0 bytes doc/integration/metrics/influxdb_configuration.md | 192 --------------------- doc/integration/metrics/influxdb_schema.md | 87 ---------- doc/integration/metrics/introduction.md | 64 ------- doc/monitoring/performance/gitlab_configuration.md | 39 +++++ .../img/metrics_gitlab_configuration_settings.png | Bin 0 -> 45148 bytes .../performance/influxdb_configuration.md | 192 +++++++++++++++++++++ doc/monitoring/performance/influxdb_schema.md | 87 ++++++++++ doc/monitoring/performance/introduction.md | 64 +++++++ 11 files changed, 383 insertions(+), 383 deletions(-) delete mode 100644 doc/integration/metrics/gitlab_configuration.md delete mode 100644 doc/integration/metrics/img/metrics_gitlab_configuration_settings.png delete mode 100644 doc/integration/metrics/influxdb_configuration.md delete mode 100644 doc/integration/metrics/influxdb_schema.md delete mode 100644 doc/integration/metrics/introduction.md create mode 100644 doc/monitoring/performance/gitlab_configuration.md create mode 100644 doc/monitoring/performance/img/metrics_gitlab_configuration_settings.png create mode 100644 doc/monitoring/performance/influxdb_configuration.md create mode 100644 doc/monitoring/performance/influxdb_schema.md create mode 100644 doc/monitoring/performance/introduction.md (limited to 'doc') diff --git a/doc/README.md b/doc/README.md index 05c0d6fa0fe..1efed2871f9 100644 --- a/doc/README.md +++ b/doc/README.md @@ -67,7 +67,7 @@ - [Reply by email](incoming_email/README.md) Allow users to comment on issues and merge requests by replying to notification emails. - [Migrate GitLab CI to CE/EE](migrate_ci_to_ce/README.md) Follow this guide to migrate your existing GitLab CI data to GitLab CE/EE. - [Git LFS configuration](workflow/lfs/lfs_administration.md) -- [GitLab Performance Monitoring](integration/metrics/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics +- [GitLab Performance Monitoring](monitoring/performance/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics ## Contributor documentation diff --git a/doc/integration/metrics/gitlab_configuration.md b/doc/integration/metrics/gitlab_configuration.md deleted file mode 100644 index b856e7935a3..00000000000 --- a/doc/integration/metrics/gitlab_configuration.md +++ /dev/null @@ -1,39 +0,0 @@ -# GitLab Configuration - -GitLab Performance Monitoring is disabled by default. To enable it and change any of its -settings, navigate to the Admin area in **Settings > Metrics** -(`/admin/application_settings`). - -The minimum required settings you need to set are the InfluxDB host and port. -Make sure _Enable InfluxDB Metrics_ is checked and hit **Save** to save the -changes. - ---- - -![GitLab Performance Monitoring Admin Settings](img/metrics_gitlab_configuration_settings.png) - ---- - -Finally, a restart of all GitLab processes is required for the changes to take -effect: - -```bash -# For Omnibus installations -sudo gitlab-ctl restart - -# For installations from source -sudo service gitlab restart -``` - -## Pending Migrations - -When any migrations are pending, the metrics are disabled until the migrations -have been performed. - ---- - -Read more on: - -- [Introduction to GitLab Performance Monitoring](introduction.md) -- [InfluxDB Configuration](influxdb_configuration.md) -- [InfluxDB Schema](influxdb_schema.md) diff --git a/doc/integration/metrics/img/metrics_gitlab_configuration_settings.png b/doc/integration/metrics/img/metrics_gitlab_configuration_settings.png deleted file mode 100644 index 14d82b6ac98..00000000000 Binary files a/doc/integration/metrics/img/metrics_gitlab_configuration_settings.png and /dev/null differ diff --git a/doc/integration/metrics/influxdb_configuration.md b/doc/integration/metrics/influxdb_configuration.md deleted file mode 100644 index df5e4af7fad..00000000000 --- a/doc/integration/metrics/influxdb_configuration.md +++ /dev/null @@ -1,192 +0,0 @@ -# InfluxDB Configuration - -The default settings provided by [InfluxDB] are not sufficient for a high traffic -GitLab environment. The settings discussed in this document are based on the -settings GitLab uses for GitLab.com, depending on your own needs you may need to -further adjust them. - -If you are intending to run InfluxDB on the same server as GitLab, make sure -you have plenty of RAM since InfluxDB can use quite a bit depending on traffic. - -Unless you are going with a budget setup, it's advised to run it separately. - -## Requirements - -- InfluxDB 0.9.5 or newer -- A fairly modern version of Linux -- At least 4GB of RAM -- At least 10GB of storage for InfluxDB data - -Note that the RAM and storage requirements can differ greatly depending on the -amount of data received/stored. To limit the amount of stored data users can -look into [InfluxDB Retention Policies][influxdb-retention]. - -## Installation - -Installing InfluxDB is out of the scope of this document. Please refer to the -[InfluxDB documentation]. - -## InfluxDB Server Settings - -Since InfluxDB has many settings that users may wish to customize themselves -(e.g. what port to run InfluxDB on), we'll only cover the essentials. - -The configuration file in question is usually located at -`/etc/influxdb/influxdb.conf`. Whenever you make a change in this file, -InfluxDB needs to be restarted. - -### Storage Engine - -InfluxDB comes with different storage engines and as of InfluxDB 0.9.5 a new -storage engine is available, called [TSM Tree]. All users **must** use the new -`tsm1` storage engine as this [will be the default engine][tsm1-commit] in -upcoming InfluxDB releases. - -Make sure you have the following in your configuration file: - -``` -[data] - dir = "/var/lib/influxdb/data" - engine = "tsm1" -``` - -### Admin Panel - -Production environments should have the InfluxDB admin panel **disabled**. This -feature can be disabled by adding the following to your InfluxDB configuration -file: - -``` -[admin] - enabled = false -``` - -### HTTP - -HTTP is required when using the [InfluxDB CLI] or other tools such as Grafana, -thus it should be enabled. When enabling make sure to _also_ enable -authentication: - -``` -[http] - enabled = true - auth-enabled = true -``` - -_**Note:** Before you enable authentication, you might want to [create an -admin user](#create-a-new-admin-user)._ - -### UDP - -GitLab writes data to InfluxDB via UDP and thus this must be enabled. Enabling -UDP can be done using the following settings: - -``` -[[udp]] - enabled = true - bind-address = ":8089" - database = "gitlab" - batch-size = 1000 - batch-pending = 5 - batch-timeout = "1s" - read-buffer = 209715200 -``` - -This does the following: - -1. Enable UDP and bind it to port 8089 for all addresses. -2. Store any data received in the "gitlab" database. -3. Define a batch of points to be 1000 points in size and allow a maximum of - 5 batches _or_ flush them automatically after 1 second. -4. Define a UDP read buffer size of 200 MB. - -One of the most important settings here is the UDP read buffer size as if this -value is set too low, packets will be dropped. You must also make sure the OS -buffer size is set to the same value, the default value is almost never enough. - -To set the OS buffer size to 200 MB, on Linux you can run the following command: - -```bash -sysctl -w net.core.rmem_max=209715200 -``` - -To make this permanent, add the following to `/etc/sysctl.conf` and restart the -server: - -```bash -net.core.rmem_max=209715200 -``` - -It is **very important** to make sure the buffer sizes are large enough to -handle all data sent to InfluxDB as otherwise you _will_ lose data. The above -buffer sizes are based on the traffic for GitLab.com. Depending on the amount of -traffic, users may be able to use a smaller buffer size, but we highly recommend -using _at least_ 100 MB. - -When enabling UDP, users should take care to not expose the port to the public, -as doing so will allow anybody to write data into your InfluxDB database (as -[InfluxDB's UDP protocol][udp] doesn't support authentication). We recommend either -whitelisting the allowed IP addresses/ranges, or setting up a VLAN and only -allowing traffic from members of said VLAN. - -## Create a new admin user - -If you want to [enable authentication](#http), you might want to [create an -admin user][influx-admin]: - -``` -influx -execute "CREATE USER thedude WITH PASSWORD '1234' WITH ALL PRIVILEGES" -``` - -## Create the `gitlab` database - -Once you get InfluxDB up and running, you need to create a database for GitLab. -Make sure you have changed the [storage engine](#storage-engine) to `tsm1` -before creating a database. - -_**Note:** If you [created an admin user](#create-a-new-admin-user) and enabled -[HTTP authentication](#http), remember to append the username (`-username thedude`) -and password (`-password 1234`) to the commands below._ - -Run the following command to create a database named `gitlab`: - -```bash -influx -execute 'CREATE DATABASE gitlab' -``` - -The name **must** be `gitlab`, do not use any other name. - -Next, make sure that the database was successfully created: - -```bash -influx -execute 'SHOW DATABASES' -``` - -The output should be similar to: - -``` -name: databases ---------------- -name -_internal -gitlab -``` - -That's it! Now your GitLab instance should send data to InfluxDB. - ---- - -Read more on: - -- [Introduction to GitLab Performance Monitoring](introduction.md) -- [GitLab Configuration](gitlab_configuration.md) -- [InfluxDB Schema](influxdb_schema.md) - -[influxdb-retention]: https://docs.influxdata.com/influxdb/v0.9/query_language/database_management/#retention-policy-management -[influxdb documentation]: https://docs.influxdata.com/influxdb/v0.9/ -[influxdb cli]: https://docs.influxdata.com/influxdb/v0.9/tools/shell/ -[udp]: https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/ -[influxdb]: https://influxdata.com/time-series-platform/influxdb/ -[tsm tree]: https://influxdata.com/blog/new-storage-engine-time-structured-merge-tree/ -[tsm1-commit]: https://github.com/influxdata/influxdb/commit/15d723dc77651bac83e09e2b1c94be480966cb0d -[influx-admin]: https://docs.influxdata.com/influxdb/v0.9/administration/authentication_and_authorization/#create-a-new-admin-user diff --git a/doc/integration/metrics/influxdb_schema.md b/doc/integration/metrics/influxdb_schema.md deleted file mode 100644 index a5a8aebd2d1..00000000000 --- a/doc/integration/metrics/influxdb_schema.md +++ /dev/null @@ -1,87 +0,0 @@ -# InfluxDB Schema - -The following measurements are currently stored in InfluxDB: - -- `PROCESS_file_descriptors` -- `PROCESS_gc_statistics` -- `PROCESS_memory_usage` -- `PROCESS_method_calls` -- `PROCESS_object_counts` -- `PROCESS_transactions` -- `PROCESS_views` - -Here, `PROCESS` is replaced with either `rails` or `sidekiq` depending on the -process type. In all series, any form of duration is stored in milliseconds. - -## PROCESS_file_descriptors - -This measurement contains the number of open file descriptors over time. The -value field `value` contains the number of descriptors. - -## PROCESS_gc_statistics - -This measurement contains Ruby garbage collection statistics such as the amount -of minor/major GC runs (relative to the last sampling interval), the time spent -in garbage collection cycles, and all fields/values returned by `GC.stat`. - -## PROCESS_memory_usage - -This measurement contains the process' memory usage (in bytes) over time. The -value field `value` contains the number of bytes. - -## PROCESS_method_calls - -This measurement contains the methods called during a transaction along with -their duration, and a name of the transaction action that invoked the method (if -available). The method call duration is stored in the value field `duration`, -while the method name is stored in the tag `method`. The tag `action` contains -the full name of the transaction action. Both the `method` and `action` fields -are in the following format: - -``` -ClassName#method_name -``` - -For example, a method called by the `show` method in the `UsersController` class -would have `action` set to `UsersController#show`. - -## PROCESS_object_counts - -This measurement is used to store retained Ruby objects (per class) and the -amount of retained objects. The number of objects is stored in the `count` value -field while the class name is stored in the `type` tag. - -## PROCESS_transactions - -This measurement is used to store basic transaction details such as the time it -took to complete a transaction, how much time was spent in SQL queries, etc. The -following value fields are available: - -| Value | Description | -| ----- | ----------- | -| `duration` | The total duration of the transaction | -| `allocated_memory` | The amount of bytes allocated while the transaction was running. This value is only reliable when using single-threaded application servers | -| `method_duration` | The total time spent in method calls | -| `sql_duration` | The total time spent in SQL queries | -| `view_duration` | The total time spent in views | - -## PROCESS_views - -This measurement is used to store view rendering timings for a transaction. The -following value fields are available: - -| Value | Description | -| ----- | ----------- | -| `duration` | The rendering time of the view | -| `view` | The path of the view, relative to the application's root directory | - -The `action` tag contains the action name of the transaction that rendered the -view. - ---- - -Read more on: - -- [Introduction to GitLab Performance Monitoring](introduction.md) -- [GitLab Configuration](gitlab_configuration.md) -- [InfluxDB Configuration](influxdb_configuration.md) diff --git a/doc/integration/metrics/introduction.md b/doc/integration/metrics/introduction.md deleted file mode 100644 index f2460d31302..00000000000 --- a/doc/integration/metrics/introduction.md +++ /dev/null @@ -1,64 +0,0 @@ -# GitLab Performance Monitoring - -GitLab comes with its own application performance measuring system as of GitLab -8.4, simply called "GitLab Performance Monitoring". GitLab Performance Monitoring is available in both the -Community and Enterprise editions. - -Apart from this introduction, you are advised to read through the following -documents in order to understand and properly configure GitLab Performance Monitoring: - -- [GitLab Configuration](gitlab_configuration.md) -- [InfluxDB Configuration](influxdb_configuration.md) -- [InfluxDB Schema](influxdb_schema.md) - -## Introduction to GitLab Performance Monitoring - -GitLab Performance Monitoring makes it possible to measure a wide variety of statistics -including (but not limited to): - -- The time it took to complete a transaction (a web request or Sidekiq job). -- The time spent in running SQL queries and rendering HAML views. -- The time spent executing (instrumented) Ruby methods. -- Ruby object allocations, and retained objects in particular. -- System statistics such as the process' memory usage and open file descriptors. -- Ruby garbage collection statistics. - -Metrics data is written to [InfluxDB][influxdb] over [UDP][influxdb-udp]. Stored -data can be visualized using [Grafana][grafana] or any other application that -supports reading data from InfluxDB. Alternatively data can be queried using the -InfluxDB CLI. - -## Metric Types - -Two types of metrics are collected: - -1. Transaction specific metrics. -1. Sampled metrics, collected at a certain interval in a separate thread. - -### Transaction Metrics - -Transaction metrics are metrics that can be associated with a single -transaction. This includes statistics such as the transaction duration, timings -of any executed SQL queries, time spent rendering HAML views, etc. These metrics -are collected for every Rack request and Sidekiq job processed. - -### Sampled Metrics - -Sampled metrics are metrics that can't be associated with a single transaction. -Examples include garbage collection statistics and retained Ruby objects. These -metrics are collected at a regular interval. This interval is made up out of two -parts: - -1. A user defined interval. -1. A randomly generated offset added on top of the interval, the same offset - can't be used twice in a row. - -The actual interval can be anywhere between a half of the defined interval and a -half above the interval. For example, for a user defined interval of 15 seconds -the actual interval can be anywhere between 7.5 and 22.5. The interval is -re-generated for every sampling run instead of being generated once and re-used -for the duration of the process' lifetime. - -[influxdb]: https://influxdata.com/time-series-platform/influxdb/ -[influxdb-udp]: https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/ -[grafana]: http://grafana.org/ diff --git a/doc/monitoring/performance/gitlab_configuration.md b/doc/monitoring/performance/gitlab_configuration.md new file mode 100644 index 00000000000..b856e7935a3 --- /dev/null +++ b/doc/monitoring/performance/gitlab_configuration.md @@ -0,0 +1,39 @@ +# GitLab Configuration + +GitLab Performance Monitoring is disabled by default. To enable it and change any of its +settings, navigate to the Admin area in **Settings > Metrics** +(`/admin/application_settings`). + +The minimum required settings you need to set are the InfluxDB host and port. +Make sure _Enable InfluxDB Metrics_ is checked and hit **Save** to save the +changes. + +--- + +![GitLab Performance Monitoring Admin Settings](img/metrics_gitlab_configuration_settings.png) + +--- + +Finally, a restart of all GitLab processes is required for the changes to take +effect: + +```bash +# For Omnibus installations +sudo gitlab-ctl restart + +# For installations from source +sudo service gitlab restart +``` + +## Pending Migrations + +When any migrations are pending, the metrics are disabled until the migrations +have been performed. + +--- + +Read more on: + +- [Introduction to GitLab Performance Monitoring](introduction.md) +- [InfluxDB Configuration](influxdb_configuration.md) +- [InfluxDB Schema](influxdb_schema.md) diff --git a/doc/monitoring/performance/img/metrics_gitlab_configuration_settings.png b/doc/monitoring/performance/img/metrics_gitlab_configuration_settings.png new file mode 100644 index 00000000000..14d82b6ac98 Binary files /dev/null and b/doc/monitoring/performance/img/metrics_gitlab_configuration_settings.png differ diff --git a/doc/monitoring/performance/influxdb_configuration.md b/doc/monitoring/performance/influxdb_configuration.md new file mode 100644 index 00000000000..df5e4af7fad --- /dev/null +++ b/doc/monitoring/performance/influxdb_configuration.md @@ -0,0 +1,192 @@ +# InfluxDB Configuration + +The default settings provided by [InfluxDB] are not sufficient for a high traffic +GitLab environment. The settings discussed in this document are based on the +settings GitLab uses for GitLab.com, depending on your own needs you may need to +further adjust them. + +If you are intending to run InfluxDB on the same server as GitLab, make sure +you have plenty of RAM since InfluxDB can use quite a bit depending on traffic. + +Unless you are going with a budget setup, it's advised to run it separately. + +## Requirements + +- InfluxDB 0.9.5 or newer +- A fairly modern version of Linux +- At least 4GB of RAM +- At least 10GB of storage for InfluxDB data + +Note that the RAM and storage requirements can differ greatly depending on the +amount of data received/stored. To limit the amount of stored data users can +look into [InfluxDB Retention Policies][influxdb-retention]. + +## Installation + +Installing InfluxDB is out of the scope of this document. Please refer to the +[InfluxDB documentation]. + +## InfluxDB Server Settings + +Since InfluxDB has many settings that users may wish to customize themselves +(e.g. what port to run InfluxDB on), we'll only cover the essentials. + +The configuration file in question is usually located at +`/etc/influxdb/influxdb.conf`. Whenever you make a change in this file, +InfluxDB needs to be restarted. + +### Storage Engine + +InfluxDB comes with different storage engines and as of InfluxDB 0.9.5 a new +storage engine is available, called [TSM Tree]. All users **must** use the new +`tsm1` storage engine as this [will be the default engine][tsm1-commit] in +upcoming InfluxDB releases. + +Make sure you have the following in your configuration file: + +``` +[data] + dir = "/var/lib/influxdb/data" + engine = "tsm1" +``` + +### Admin Panel + +Production environments should have the InfluxDB admin panel **disabled**. This +feature can be disabled by adding the following to your InfluxDB configuration +file: + +``` +[admin] + enabled = false +``` + +### HTTP + +HTTP is required when using the [InfluxDB CLI] or other tools such as Grafana, +thus it should be enabled. When enabling make sure to _also_ enable +authentication: + +``` +[http] + enabled = true + auth-enabled = true +``` + +_**Note:** Before you enable authentication, you might want to [create an +admin user](#create-a-new-admin-user)._ + +### UDP + +GitLab writes data to InfluxDB via UDP and thus this must be enabled. Enabling +UDP can be done using the following settings: + +``` +[[udp]] + enabled = true + bind-address = ":8089" + database = "gitlab" + batch-size = 1000 + batch-pending = 5 + batch-timeout = "1s" + read-buffer = 209715200 +``` + +This does the following: + +1. Enable UDP and bind it to port 8089 for all addresses. +2. Store any data received in the "gitlab" database. +3. Define a batch of points to be 1000 points in size and allow a maximum of + 5 batches _or_ flush them automatically after 1 second. +4. Define a UDP read buffer size of 200 MB. + +One of the most important settings here is the UDP read buffer size as if this +value is set too low, packets will be dropped. You must also make sure the OS +buffer size is set to the same value, the default value is almost never enough. + +To set the OS buffer size to 200 MB, on Linux you can run the following command: + +```bash +sysctl -w net.core.rmem_max=209715200 +``` + +To make this permanent, add the following to `/etc/sysctl.conf` and restart the +server: + +```bash +net.core.rmem_max=209715200 +``` + +It is **very important** to make sure the buffer sizes are large enough to +handle all data sent to InfluxDB as otherwise you _will_ lose data. The above +buffer sizes are based on the traffic for GitLab.com. Depending on the amount of +traffic, users may be able to use a smaller buffer size, but we highly recommend +using _at least_ 100 MB. + +When enabling UDP, users should take care to not expose the port to the public, +as doing so will allow anybody to write data into your InfluxDB database (as +[InfluxDB's UDP protocol][udp] doesn't support authentication). We recommend either +whitelisting the allowed IP addresses/ranges, or setting up a VLAN and only +allowing traffic from members of said VLAN. + +## Create a new admin user + +If you want to [enable authentication](#http), you might want to [create an +admin user][influx-admin]: + +``` +influx -execute "CREATE USER thedude WITH PASSWORD '1234' WITH ALL PRIVILEGES" +``` + +## Create the `gitlab` database + +Once you get InfluxDB up and running, you need to create a database for GitLab. +Make sure you have changed the [storage engine](#storage-engine) to `tsm1` +before creating a database. + +_**Note:** If you [created an admin user](#create-a-new-admin-user) and enabled +[HTTP authentication](#http), remember to append the username (`-username thedude`) +and password (`-password 1234`) to the commands below._ + +Run the following command to create a database named `gitlab`: + +```bash +influx -execute 'CREATE DATABASE gitlab' +``` + +The name **must** be `gitlab`, do not use any other name. + +Next, make sure that the database was successfully created: + +```bash +influx -execute 'SHOW DATABASES' +``` + +The output should be similar to: + +``` +name: databases +--------------- +name +_internal +gitlab +``` + +That's it! Now your GitLab instance should send data to InfluxDB. + +--- + +Read more on: + +- [Introduction to GitLab Performance Monitoring](introduction.md) +- [GitLab Configuration](gitlab_configuration.md) +- [InfluxDB Schema](influxdb_schema.md) + +[influxdb-retention]: https://docs.influxdata.com/influxdb/v0.9/query_language/database_management/#retention-policy-management +[influxdb documentation]: https://docs.influxdata.com/influxdb/v0.9/ +[influxdb cli]: https://docs.influxdata.com/influxdb/v0.9/tools/shell/ +[udp]: https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/ +[influxdb]: https://influxdata.com/time-series-platform/influxdb/ +[tsm tree]: https://influxdata.com/blog/new-storage-engine-time-structured-merge-tree/ +[tsm1-commit]: https://github.com/influxdata/influxdb/commit/15d723dc77651bac83e09e2b1c94be480966cb0d +[influx-admin]: https://docs.influxdata.com/influxdb/v0.9/administration/authentication_and_authorization/#create-a-new-admin-user diff --git a/doc/monitoring/performance/influxdb_schema.md b/doc/monitoring/performance/influxdb_schema.md new file mode 100644 index 00000000000..a5a8aebd2d1 --- /dev/null +++ b/doc/monitoring/performance/influxdb_schema.md @@ -0,0 +1,87 @@ +# InfluxDB Schema + +The following measurements are currently stored in InfluxDB: + +- `PROCESS_file_descriptors` +- `PROCESS_gc_statistics` +- `PROCESS_memory_usage` +- `PROCESS_method_calls` +- `PROCESS_object_counts` +- `PROCESS_transactions` +- `PROCESS_views` + +Here, `PROCESS` is replaced with either `rails` or `sidekiq` depending on the +process type. In all series, any form of duration is stored in milliseconds. + +## PROCESS_file_descriptors + +This measurement contains the number of open file descriptors over time. The +value field `value` contains the number of descriptors. + +## PROCESS_gc_statistics + +This measurement contains Ruby garbage collection statistics such as the amount +of minor/major GC runs (relative to the last sampling interval), the time spent +in garbage collection cycles, and all fields/values returned by `GC.stat`. + +## PROCESS_memory_usage + +This measurement contains the process' memory usage (in bytes) over time. The +value field `value` contains the number of bytes. + +## PROCESS_method_calls + +This measurement contains the methods called during a transaction along with +their duration, and a name of the transaction action that invoked the method (if +available). The method call duration is stored in the value field `duration`, +while the method name is stored in the tag `method`. The tag `action` contains +the full name of the transaction action. Both the `method` and `action` fields +are in the following format: + +``` +ClassName#method_name +``` + +For example, a method called by the `show` method in the `UsersController` class +would have `action` set to `UsersController#show`. + +## PROCESS_object_counts + +This measurement is used to store retained Ruby objects (per class) and the +amount of retained objects. The number of objects is stored in the `count` value +field while the class name is stored in the `type` tag. + +## PROCESS_transactions + +This measurement is used to store basic transaction details such as the time it +took to complete a transaction, how much time was spent in SQL queries, etc. The +following value fields are available: + +| Value | Description | +| ----- | ----------- | +| `duration` | The total duration of the transaction | +| `allocated_memory` | The amount of bytes allocated while the transaction was running. This value is only reliable when using single-threaded application servers | +| `method_duration` | The total time spent in method calls | +| `sql_duration` | The total time spent in SQL queries | +| `view_duration` | The total time spent in views | + +## PROCESS_views + +This measurement is used to store view rendering timings for a transaction. The +following value fields are available: + +| Value | Description | +| ----- | ----------- | +| `duration` | The rendering time of the view | +| `view` | The path of the view, relative to the application's root directory | + +The `action` tag contains the action name of the transaction that rendered the +view. + +--- + +Read more on: + +- [Introduction to GitLab Performance Monitoring](introduction.md) +- [GitLab Configuration](gitlab_configuration.md) +- [InfluxDB Configuration](influxdb_configuration.md) diff --git a/doc/monitoring/performance/introduction.md b/doc/monitoring/performance/introduction.md new file mode 100644 index 00000000000..f2460d31302 --- /dev/null +++ b/doc/monitoring/performance/introduction.md @@ -0,0 +1,64 @@ +# GitLab Performance Monitoring + +GitLab comes with its own application performance measuring system as of GitLab +8.4, simply called "GitLab Performance Monitoring". GitLab Performance Monitoring is available in both the +Community and Enterprise editions. + +Apart from this introduction, you are advised to read through the following +documents in order to understand and properly configure GitLab Performance Monitoring: + +- [GitLab Configuration](gitlab_configuration.md) +- [InfluxDB Configuration](influxdb_configuration.md) +- [InfluxDB Schema](influxdb_schema.md) + +## Introduction to GitLab Performance Monitoring + +GitLab Performance Monitoring makes it possible to measure a wide variety of statistics +including (but not limited to): + +- The time it took to complete a transaction (a web request or Sidekiq job). +- The time spent in running SQL queries and rendering HAML views. +- The time spent executing (instrumented) Ruby methods. +- Ruby object allocations, and retained objects in particular. +- System statistics such as the process' memory usage and open file descriptors. +- Ruby garbage collection statistics. + +Metrics data is written to [InfluxDB][influxdb] over [UDP][influxdb-udp]. Stored +data can be visualized using [Grafana][grafana] or any other application that +supports reading data from InfluxDB. Alternatively data can be queried using the +InfluxDB CLI. + +## Metric Types + +Two types of metrics are collected: + +1. Transaction specific metrics. +1. Sampled metrics, collected at a certain interval in a separate thread. + +### Transaction Metrics + +Transaction metrics are metrics that can be associated with a single +transaction. This includes statistics such as the transaction duration, timings +of any executed SQL queries, time spent rendering HAML views, etc. These metrics +are collected for every Rack request and Sidekiq job processed. + +### Sampled Metrics + +Sampled metrics are metrics that can't be associated with a single transaction. +Examples include garbage collection statistics and retained Ruby objects. These +metrics are collected at a regular interval. This interval is made up out of two +parts: + +1. A user defined interval. +1. A randomly generated offset added on top of the interval, the same offset + can't be used twice in a row. + +The actual interval can be anywhere between a half of the defined interval and a +half above the interval. For example, for a user defined interval of 15 seconds +the actual interval can be anywhere between 7.5 and 22.5. The interval is +re-generated for every sampling run instead of being generated once and re-used +for the duration of the process' lifetime. + +[influxdb]: https://influxdata.com/time-series-platform/influxdb/ +[influxdb-udp]: https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/ +[grafana]: http://grafana.org/ -- cgit v1.2.1 From dd59fc213c6cf2a2efb39eda241f453bbd1e82c7 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 20 Jan 2016 23:10:27 +0100 Subject: Change InfluxDB admin username [ci skip] --- doc/monitoring/performance/influxdb_configuration.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) (limited to 'doc') diff --git a/doc/monitoring/performance/influxdb_configuration.md b/doc/monitoring/performance/influxdb_configuration.md index df5e4af7fad..3a2b598b78f 100644 --- a/doc/monitoring/performance/influxdb_configuration.md +++ b/doc/monitoring/performance/influxdb_configuration.md @@ -135,7 +135,7 @@ If you want to [enable authentication](#http), you might want to [create an admin user][influx-admin]: ``` -influx -execute "CREATE USER thedude WITH PASSWORD '1234' WITH ALL PRIVILEGES" +influx -execute "CREATE USER jeff WITH PASSWORD '1234' WITH ALL PRIVILEGES" ``` ## Create the `gitlab` database @@ -145,8 +145,8 @@ Make sure you have changed the [storage engine](#storage-engine) to `tsm1` before creating a database. _**Note:** If you [created an admin user](#create-a-new-admin-user) and enabled -[HTTP authentication](#http), remember to append the username (`-username thedude`) -and password (`-password 1234`) to the commands below._ +[HTTP authentication](#http), remember to append the username (`-username `) +and password (`-password `) you set earlier to the commands below._ Run the following command to create a database named `gitlab`: -- cgit v1.2.1