summaryrefslogtreecommitdiff
path: root/doc/monitoring/performance
diff options
context:
space:
mode:
authorDrew Blessing <drew@gitlab.com>2016-02-29 14:21:37 -0600
committerDrew Blessing <drew@gitlab.com>2016-03-22 13:34:05 -0500
commit1d99c8fe603e8c3b80b59aedbfa5e31a51ce61b2 (patch)
treed231a6d5bd425e7fe4e09f7a8027235165647baf /doc/monitoring/performance
parentf27210d54b98ac6e3113369b6a94c50ddb5f66a5 (diff)
downloadgitlab-ce-1d99c8fe603e8c3b80b59aedbfa5e31a51ce61b2.tar.gz
Grafana installation and configuration documentation. [ci skip]
Diffstat (limited to 'doc/monitoring/performance')
-rw-r--r--doc/monitoring/performance/grafana_configuration.md118
-rw-r--r--doc/monitoring/performance/img/grafana_dashboard_dropdown.pngbin0 -> 29419 bytes
-rw-r--r--doc/monitoring/performance/img/grafana_dashboard_import.pngbin0 -> 40974 bytes
-rw-r--r--doc/monitoring/performance/img/grafana_data_source_configuration.pngbin0 -> 53402 bytes
-rw-r--r--doc/monitoring/performance/img/grafana_data_source_empty.pngbin0 -> 44058 bytes
-rw-r--r--doc/monitoring/performance/img/grafana_save_icon.pngbin0 -> 16024 bytes
-rw-r--r--doc/monitoring/performance/introduction.md3
7 files changed, 120 insertions, 1 deletions
diff --git a/doc/monitoring/performance/grafana_configuration.md b/doc/monitoring/performance/grafana_configuration.md
new file mode 100644
index 00000000000..416c9870aa0
--- /dev/null
+++ b/doc/monitoring/performance/grafana_configuration.md
@@ -0,0 +1,118 @@
+# Grafana Configuration
+
+[Grafana](http://grafana.org/) is a tool that allows you to visualize time
+series metrics through graphs and dashboards. It supports several backend
+data stores, including InfluxDB. GitLab writes performance data to InfluxDB
+and Grafana will allow you to query InfluxDB to display useful graphs.
+
+For the easiest installation and configuration, install Grafana on the same
+server as InfluxDB. For larger installations, you may want to split out these
+services.
+
+## Installation
+
+Grafana supplies package repositories (Yum/Apt) for easy installation.
+See [Grafana installation documentation](http://docs.grafana.org/installation/)
+for detailed steps.
+
+> **Note**: Before starting Grafana for the first time, set the admin user
+and password in `/etc/grafana/grafana.ini`. Otherwise, the default password
+will be `admin`.
+
+## Configuration
+
+Login as the admin user. Expand the menu by clicking the Grafana logo in the
+top left corner. Choose 'Data Sources' from the menu. Then, click 'Add new'
+in the top bar.
+
+![Grafana empty data source page](img/grafana_data_source_empty.png)
+
+Fill in the configuration details for the InfluxDB data source. Save and
+Test Connection to ensure the configuration is correct.
+
+- **Name**: InfluxDB
+- **Default**: Checked
+- **Type**: InfluxDB 0.9.x (Even if you're using InfluxDB 0.10.x)
+- **Url**: https://localhost:8086 (Or the remote URL if you've installed InfluxDB
+on a separate server)
+- **Access**: proxy
+- **Database**: gitlab
+- **User**: admin (Or the username configured when setting up InfluxDB)
+- **Password**: The password configured when you set up InfluxDB
+
+![Grafana data source configurations](img/grafana_data_source_configuration.png)
+
+## Apply retention policies and create continuous queries
+
+If you intend to import the GitLab provided Grafana dashboards, you will need
+to copy and run a set of queries against InfluxDB to create the needed data
+sets.
+
+On the InfluxDB server, run the following command, substituting your InfluxDB
+user and password:
+
+```bash
+influxdb --username admin -password super_secret
+```
+
+This will drop you in to an InfluxDB interactive session. Copy the entire
+contents below and paste it in to the interactive session:
+
+```
+CREATE RETENTION POLICY gitlab_30d ON gitlab DURATION 30d REPLICATION 1 DEFAULT
+CREATE RETENTION POLICY seven_days ON gitlab DURATION 7d REPLICATION 1
+CREATE CONTINUOUS QUERY rails_transaction_timings_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS "duration_95th", percentile("duration", 99.000) AS "duration_99th", mean("duration") AS "duration_mean", percentile(sql_duration, 95.000) AS "sql_duration_95th", percentile(sql_duration, 99.000) AS "sql_duration_99th", mean(sql_duration) AS "sql_duration_mean", percentile(view_duration, 95.000) AS "view_duration_95th", percentile(view_duration, 99.000) AS "view_duration_99th", mean(view_duration) AS "view_duration_mean" INTO gitlab.seven_days.rails_transaction_timings FROM gitlab.gitlab_30d.rails_transactions GROUP BY time(1m) END
+CREATE CONTINUOUS QUERY sidekiq_transaction_timings_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS "duration_95th", percentile("duration", 99.000) AS "duration_99th", mean("duration") AS "duration_mean", percentile(sql_duration, 95.000) AS "sql_duration_95th", percentile(sql_duration, 99.000) AS "sql_duration_99th", mean(sql_duration) AS "sql_duration_mean", percentile(view_duration, 95.000) AS "view_duration_95th", percentile(view_duration, 99.000) AS "view_duration_99th", mean(view_duration) AS "view_duration_mean" INTO gitlab.seven_days.sidekiq_transaction_timings FROM gitlab.gitlab_30d.sidekiq_transactions GROUP BY time(1m) END
+CREATE CONTINUOUS QUERY rails_transaction_counts_seven_days ON gitlab BEGIN SELECT count("duration") AS "count" INTO gitlab.seven_days.rails_transaction_counts FROM gitlab.gitlab_30d.rails_transactions GROUP BY time(1m) END
+CREATE CONTINUOUS QUERY sidekiq_transaction_counts_seven_days ON gitlab BEGIN SELECT count("duration") AS "count" INTO gitlab.seven_days.sidekiq_transaction_counts FROM gitlab.gitlab_30d.sidekiq_transactions GROUP BY time(1m) END
+CREATE CONTINUOUS QUERY rails_method_call_timings_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS "duration_95th", percentile("duration", 99.000) AS "duration_99th", mean("duration") AS "duration_mean" INTO gitlab.seven_days.rails_method_call_timings FROM gitlab.gitlab_30d.rails_method_calls GROUP BY time(1m) END
+CREATE CONTINUOUS QUERY sidekiq_method_call_timings_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS "duration_95th", percentile("duration", 99.000) AS "duration_99th", mean("duration") AS "duration_mean" INTO gitlab.seven_days.sidekiq_method_call_timings FROM gitlab.gitlab_30d.sidekiq_method_calls GROUP BY time(1m) END
+CREATE CONTINUOUS QUERY rails_method_call_timings_per_method_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS duration_95th, percentile("duration", 99.000) AS duration_99th, mean("duration") AS duration_mean INTO gitlab.seven_days.rails_method_call_timings_per_method FROM gitlab.gitlab_30d.rails_method_calls GROUP BY time(1m), method END
+CREATE CONTINUOUS QUERY sidekiq_method_call_timings_per_method_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS duration_95th, percentile("duration", 99.000) AS duration_99th, mean("duration") AS duration_mean INTO gitlab.seven_days.sidekiq_method_call_timings_per_method FROM gitlab.gitlab_30d.sidekiq_method_calls GROUP BY time(1m), method END
+CREATE CONTINUOUS QUERY rails_memory_usage_per_minute ON gitlab BEGIN SELECT percentile(value, 95.000) AS memory_95th, percentile(value, 99.000) AS memory_99th, mean(value) AS memory_mean INTO gitlab.seven_days.rails_memory_usage_per_minute FROM gitlab.gitlab_30d.rails_memory_usage GROUP BY time(1m) END
+CREATE CONTINUOUS QUERY sidekiq_memory_usage_per_minute ON gitlab BEGIN SELECT percentile(value, 95.000) AS memory_95th, percentile(value, 99.000) AS memory_99th, mean(value) AS memory_mean INTO gitlab.seven_days.sidekiq_memory_usage_per_minute FROM gitlab.gitlab_30d.sidekiq_memory_usage GROUP BY time(1m) END
+CREATE CONTINUOUS QUERY sidekiq_file_descriptors_per_minute ON gitlab BEGIN SELECT sum(value) AS value INTO gitlab.seven_days.sidekiq_file_descriptors_per_minute FROM gitlab.gitlab_30d.sidekiq_file_descriptors GROUP BY time(1m) END
+CREATE CONTINUOUS QUERY rails_file_descriptors_per_minute ON gitlab BEGIN SELECT sum(value) AS value INTO gitlab.seven_days.rails_file_descriptors_per_minute FROM gitlab.gitlab_30d.rails_file_descriptors GROUP BY time(1m) END
+CREATE CONTINUOUS QUERY rails_gc_counts_per_minute ON gitlab BEGIN SELECT sum(count) AS count INTO gitlab.seven_days.rails_gc_counts_per_minute FROM gitlab.gitlab_30d.rails_gc_statistics GROUP BY time(1m) END
+CREATE CONTINUOUS QUERY sidekiq_gc_counts_per_minute ON gitlab BEGIN SELECT sum(count) AS count INTO gitlab.seven_days.sidekiq_gc_counts_per_minute FROM gitlab.gitlab_30d.sidekiq_gc_statistics GROUP BY time(1m) END
+CREATE CONTINUOUS QUERY rails_gc_timings_per_minute ON gitlab BEGIN SELECT percentile(total_time, 95.000) AS duration_95th, percentile(total_time, 99.000) AS duration_99th, mean(total_time) AS duration_mean INTO gitlab.seven_days.rails_gc_timings_per_minute FROM gitlab.gitlab_30d.rails_gc_statistics GROUP BY time(1m) END
+CREATE CONTINUOUS QUERY sidekiq_gc_timings_per_minute ON gitlab BEGIN SELECT percentile(total_time, 95.000) AS duration_95th, percentile(total_time, 99.000) AS duration_99th, mean(total_time) AS duration_mean INTO gitlab.seven_days.sidekiq_gc_timings_per_minute FROM gitlab.gitlab_30d.sidekiq_gc_statistics GROUP BY time(1m) END
+CREATE CONTINUOUS QUERY rails_gc_major_minor_per_minute ON gitlab BEGIN SELECT sum(major_gc_count) AS major, sum(minor_gc_count) AS minor INTO gitlab.seven_days.rails_gc_major_minor_per_minute FROM gitlab.gitlab_30d.rails_gc_statistics GROUP BY time(1m) END
+CREATE CONTINUOUS QUERY sidekiq_gc_major_minor_per_minute ON gitlab BEGIN SELECT sum(major_gc_count) AS major, sum(minor_gc_count) AS minor INTO gitlab.seven_days.sidekiq_gc_major_minor_per_minute FROM gitlab.gitlab_30d.sidekiq_gc_statistics GROUP BY time(1m) END
+```
+
+## Import Dashboards
+
+You can now import a set of default dashboards that will give you a good
+start on displaying useful information. GitLab has published a set of default
+[Grafana dashboards][grafana-dashboards] to get you started. Clone the
+repository or download a zip/tarball, then follow these steps to import each
+JSON file.
+
+Open the dashboard dropdown menu and click 'Import'
+
+![Grafana dashboard dropdown](/img/grafana_dashboard_dropdown.png)
+
+Click 'Choose file' and browse to the location where you downloaded or cloned
+the dashboard repository. Pick one of the JSON files to import.
+
+![Grafana dashboard import](/img/grafana_dashboard_import.png)
+
+Once the dashboard is imported, be sure to click save icon in the top bar. If
+you do not save the dashboard after importing it will be removed when you
+navigate away.
+
+![Grafana save icon](/img/grafana_save_icon.png)
+
+Repeat this process for each dashboard you wish to import.
+
+[grafana-dashboards]: https://gitlab.com/gitlab-org/grafana-dashboards
+
+---
+
+Read more on:
+
+- [Introduction to GitLab Performance Monitoring](introduction.md)
+- [GitLab Configuration](gitlab_configuration.md)
+- [InfluxDB Installation/Configuration](influxdb_configuration.md)
+- [InfluxDB Schema](influxdb_schema.md)
diff --git a/doc/monitoring/performance/img/grafana_dashboard_dropdown.png b/doc/monitoring/performance/img/grafana_dashboard_dropdown.png
new file mode 100644
index 00000000000..b4448c7a09f
--- /dev/null
+++ b/doc/monitoring/performance/img/grafana_dashboard_dropdown.png
Binary files differ
diff --git a/doc/monitoring/performance/img/grafana_dashboard_import.png b/doc/monitoring/performance/img/grafana_dashboard_import.png
new file mode 100644
index 00000000000..5a2d3c0937a
--- /dev/null
+++ b/doc/monitoring/performance/img/grafana_dashboard_import.png
Binary files differ
diff --git a/doc/monitoring/performance/img/grafana_data_source_configuration.png b/doc/monitoring/performance/img/grafana_data_source_configuration.png
new file mode 100644
index 00000000000..7e2e111f570
--- /dev/null
+++ b/doc/monitoring/performance/img/grafana_data_source_configuration.png
Binary files differ
diff --git a/doc/monitoring/performance/img/grafana_data_source_empty.png b/doc/monitoring/performance/img/grafana_data_source_empty.png
new file mode 100644
index 00000000000..11e27571e64
--- /dev/null
+++ b/doc/monitoring/performance/img/grafana_data_source_empty.png
Binary files differ
diff --git a/doc/monitoring/performance/img/grafana_save_icon.png b/doc/monitoring/performance/img/grafana_save_icon.png
new file mode 100644
index 00000000000..3d4265bee8e
--- /dev/null
+++ b/doc/monitoring/performance/img/grafana_save_icon.png
Binary files differ
diff --git a/doc/monitoring/performance/introduction.md b/doc/monitoring/performance/introduction.md
index f2460d31302..79904916b7e 100644
--- a/doc/monitoring/performance/introduction.md
+++ b/doc/monitoring/performance/introduction.md
@@ -8,8 +8,9 @@ Apart from this introduction, you are advised to read through the following
documents in order to understand and properly configure GitLab Performance Monitoring:
- [GitLab Configuration](gitlab_configuration.md)
-- [InfluxDB Configuration](influxdb_configuration.md)
+- [InfluxDB Install/Configuration](influxdb_configuration.md)
- [InfluxDB Schema](influxdb_schema.md)
+- [Grafana Install/Configuration](grafana_configuration.md)
## Introduction to GitLab Performance Monitoring