Squashed commit of the following:

commit 22e1cb8f4b98f71d21026f69aa3e68d79946d6ae
Merge: 766a42a9639 e966c6aea25
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Fri Feb 23 21:30:46 2018 +0100

    Merge remote-tracking branch 'upstream/master' into 38783-add-cluster-metrics.yml

    # Conflicts:
    #	app/controllers/projects/prometheus/metrics_controller.rb
    #	app/controllers/projects/prometheus_controller.rb
    #	app/models/project_services/prometheus_service.rb
    #	lib/gitlab/prometheus/queries/query_additional_metrics.rb
    #	spec/controllers/projects/prometheus/metrics_controller_spec.rb
    #	spec/models/project_services/prometheus_service_spec.rb

commit 766a42a96393f502d439c1f0beb0b6cfb2c228d4
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Fri Feb 23 17:58:45 2018 +0100

    Move prometheus adapter to app/models/concerns

commit ca84eed49811cf3064a2e5ea611af1c947c590d7
Merge: 66702099586 cb504cedc2a
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Fri Feb 23 15:36:03 2018 +0100

    Merge remote-tracking branch 'upstream/backport_custom_metrics_ce_components' into 38783-add-cluster-metrics.yml

    + fix failing tests

    # Conflicts:
    #	app/controllers/projects/prometheus_controller.rb
    #	app/models/project_services/prometheus_service.rb
    #	lib/gitlab/prometheus/queries/query_additional_metrics.rb
    #	spec/models/project_services/prometheus_service_spec.rb

commit cb504cedc2a6e353ffb56833334681e3da09fc14
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Fri Feb 23 14:46:15 2018 +0100

    Fix prometheus_service found by find_or_initialize_service

commit 928b84c72c2c4c46e1785b9a943c1822a137de16
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 22 23:15:11 2018 +0100

    additional metrics and backported tests

commit 41291383b4b6976af94eaf9e1a7b2e2a172310e7
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 22 21:57:45 2018 +0100

    Introduce Metrics controller and retire prometheus controller

commit 6e7492e4c7ffa9d8621f09198071bc14be875976
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 22 21:46:37 2018 +0100

    Backport PrometheusClient::Error and all->common_metrics rename

commit 66702099586c864a7f78970a0ee0dd9f3c2beeef
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 22 20:23:03 2018 +0100

    fix failing tests

commit ce921ea723cde61b2441ba98c9aca97606c719c8
Author: Mike Greiling <mike@pixelcog.com>
Date:   Wed Feb 21 23:47:30 2018 -0600

    add labels to cluster metrics

commit 993830c6892e4fbf53d3f72da00002d642c7e9b2
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 22 01:20:49 2018 +0100

    Fix formatting probelms and few small tests

commit 66ee65d8e3f747c90d986bc3056178422156bc8e
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 22 01:08:03 2018 +0100

    stop using in deployment tests environment.id

commit 62c91978d15f0369988521363dae24bd7510d68d
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 22 01:06:33 2018 +0100

    fix prometheus_controller and adapter tests

commit 977b1d34c1d03c7233582e8328f85caf634895ed
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 22 00:19:39 2018 +0100

    finish up active? -> can_query? rename

commit e614f7daee58a9758d83ba3efe1649c8b80bc1e2
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Wed Feb 21 23:51:33 2018 +0100

    deployment prometheus adapter tests fix

commit ebd726c114a6026fef0adf3eba6ee1972530148a
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Wed Feb 21 21:40:40 2018 +0100

    Move environment dependant tests to environment

commit 6d31311cd3729c29233283dded70e03a4a9a3c97
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Wed Feb 21 20:40:24 2018 +0100

    update monitoring service and move adding dpeloyment_time to deployment model

commit 60b6bf391ab36846dce122bc6b0c5196a186267c
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Wed Feb 21 20:15:46 2018 +0100

    adjust deployment spec and prometheus specs

commit 6681662cf1c028aff2ff94aa0501732cb7119ba1
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 22:30:20 2018 +0100

    Revert changes to reactive caching

commit e282f86c45a056889f57d3f7fd23a81c88efff6a
Merge: 5751c73df59 6844a2df873
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 22:08:17 2018 +0100

    Merge remote-tracking branch 'upstream/master' into 38783-add-cluster-metrics.yml

commit 5751c73df59d0a03840a1b4b71b0637670f971a6
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 22:07:43 2018 +0100

    rename active? to can_query? and cleanup environment prometheus router

commit 3f3c6e1d33dcd9315979daf26a95f2aab83a7de9
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 21:51:43 2018 +0100

    Fix tests, and only use prometheus service if its active

commit 6345838bac584c213b665d334252ccab202cb271
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 16:24:27 2018 +0100

    Fix typo

commit 7a585d32afe8da050b5615b1d036a550e06479f5
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 15:29:40 2018 +0100

    Cluster id is not required

commit e6af62afb11fa380f6aff1c31a81bcc9bab3b1eb
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 15:15:46 2018 +0100

    Result transformation support

commit f3b1bd7c67894f44efe33591ddb70093bd620c03
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 15:07:07 2018 +0100

    Fix rubocop warning and exten cluster query timeframe

commit be77947cea64261a4d3dead33c3c57f413a9880c
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 14:42:47 2018 +0100

    Fix additional metrics test

commit eb3922e16221abe16f59fae1c38122f227643343
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 14:34:44 2018 +0100

    rename prometheus adapter methods

commit 045476cd08b21593818b274ae8a44d19b705523f
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 13:19:36 2018 +0100

    Make prometheus adapter a module

commit f2daf050d8c689f72c4c61207930bc53c331f12e
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 20 12:00:44 2018 +0100

    refactoring wip

commit 52e4ef5587794e811dc10a0f2dca522342a865da
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Sun Feb 18 20:25:55 2018 +0100

    cleanup prometheus adapter concept

commit 3887365faab9dfcd9c00bcfc501d09ac62431a03
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Sun Feb 18 19:22:11 2018 +0100

    Refactor out deployment id, Rename PrometheusQuerier to PrometheusAdapter

commit aa2fc2df57bd72c9a5e94f66d1f1e23990be6c3f
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Fri Feb 16 23:59:54 2018 +0100

    Refactor prometheus client

commit e43c1ca9d9874d6cf1569f40fa1aca158d9d5d91
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 15 20:24:45 2018 +0100

    Use initial version of cluster_metrics.yml

commit 867821ce0b2609ebf8994220aa8e3a94d66a01e0
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 15 05:24:14 2018 +0100

    Fix querying cluster metrics

commit 1601e002a064cbb10ffe110a19433c5662858f1d
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 15 04:49:03 2018 +0100

    Queues for unicersal querier

commit 5db198fdc925c0223be24939b76da1d544dd569c
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 15 03:35:17 2018 +0100

    refactor reactive caching and prometheus querying

commit b0fc00e8c9d21e961ef44b0129103e2a62928b52
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Thu Feb 15 00:12:03 2018 +0100

    Add generic query additional metrics

commit ffe76e6a9ba196bccff22c4880e0384959ad5e48
Author: Pawel Chojnacki <pawel@chojnacki.ws>
Date:   Tue Feb 13 15:40:15 2018 +0100

    Cluster Metric yml initial

+ Remove cluster query

+ remove cluster_metrics.yml

+ Prometheus adapter tests

11 files changed, 101 insertions, 108 deletions

diff --git a/app/assets/javascripts/monitoring/utils/multiple_time_series.js b/app/assets/javascripts/monitoring/utils/multiple_time_series.js
index 4ce3dad440c..b5b8e3c255d 100644
--- a/app/assets/javascripts/monitoring/utils/multiple_time_series.js
+++ b/app/assets/javascripts/monitoring/utils/multiple_time_series.js
@@ -76,7 +76,7 @@ function queryTimeSeries(query, graphWidth, graphHeight, graphHeightOffset, xDom
       metricTag = seriesCustomizationData.value || timeSeriesMetricLabel;
       [lineColor, areaColor] = pickColor(seriesCustomizationData.color);
     } else {
-      metricTag = timeSeriesMetricLabel || `series ${timeSeriesNumber + 1}`;
+      metricTag = timeSeriesMetricLabel || query.label || `series ${timeSeriesNumber + 1}`;
       [lineColor, areaColor] = pickColor();
     }
 
diff --git a/app/controllers/projects/deployments_controller.rb b/app/controllers/projects/deployments_controller.rb
index 1a418d0f15a..b68cdc39cb8 100644
--- a/app/controllers/projects/deployments_controller.rb
+++ b/app/controllers/projects/deployments_controller.rb
@@ -24,7 +24,7 @@ class Projects::DeploymentsController < Projects::ApplicationController
   end
 
   def additional_metrics
-    return render_404 unless deployment.has_additional_metrics?
+    return render_404 unless deployment.has_metrics?
 
     respond_to do |format|
       format.json do
diff --git a/app/controllers/projects/prometheus/metrics_controller.rb b/app/controllers/projects/prometheus/metrics_controller.rb
index b739d0f0f90..255fbf254fb 100644
--- a/app/controllers/projects/prometheus/metrics_controller.rb
+++ b/app/controllers/projects/prometheus/metrics_controller.rb
@@ -2,11 +2,12 @@ module Projects
   module Prometheus
     class MetricsController < Projects::ApplicationController
       before_action :authorize_admin_project!
+      before_action :require_prometheus_metrics!
 
       def active_common
         respond_to do |format|
           format.json do
-            matched_metrics = prometheus_service.matched_metrics || {}
+            matched_metrics = prometheus_adapter.query(:matched_metrics) || {}
 
             if matched_metrics.any?
               render json: matched_metrics
@@ -19,8 +20,12 @@ module Projects
 
       private
 
-      def prometheus_service
-        @prometheus_service ||= project.find_or_initialize_service('prometheus')
+      def prometheus_adapter
+        project.prometheus_service
+      end
+
+      def require_prometheus_metrics!
+        render_404 unless prometheus_adapter.can_query?
       end
     end
   end
diff --git a/app/models/clusters/applications/prometheus.rb b/app/models/clusters/applications/prometheus.rb
index aa22e9d5d58..ba63d0f3c64 100644
--- a/app/models/clusters/applications/prometheus.rb
+++ b/app/models/clusters/applications/prometheus.rb
@@ -1,6 +1,8 @@
 module Clusters
   module Applications
     class Prometheus < ActiveRecord::Base
+      include PrometheusAdapter
+
       VERSION = "2.0.0".freeze
 
       self.table_name = 'clusters_applications_prometheus'
@@ -38,7 +40,7 @@ module Clusters
         Gitlab::Kubernetes::Helm::InstallCommand.new(name, chart: chart, chart_values_file: chart_values_file)
       end
 
-      def proxy_client
+      def prometheus_client
         return unless kube_client
 
         proxy_url = kube_client.proxy_url('service', service_name, service_port, Gitlab::Kubernetes::Helm::NAMESPACE)
diff --git a/app/models/clusters/cluster.rb b/app/models/clusters/cluster.rb
index 8678f70f78c..1fd7db30107 100644
--- a/app/models/clusters/cluster.rb
+++ b/app/models/clusters/cluster.rb
@@ -50,7 +50,6 @@ module Clusters
     scope :disabled, -> { where(enabled: false) }
 
     scope :for_environment, -> (env) { where(environment_scope: ['*', '', env.slug]) }
-    scope :for_all_environments, -> { where(environment_scope: ['*', '']) }
 
     def status_name
       if provider
diff --git a/app/models/clusters/platforms/kubernetes.rb b/app/models/clusters/platforms/kubernetes.rb
index 7ce8befeeeb..66f629df1b4 100644
--- a/app/models/clusters/platforms/kubernetes.rb
+++ b/app/models/clusters/platforms/kubernetes.rb
@@ -141,7 +141,7 @@ module Clusters
       end
 
       def kubeclient_ssl_options
-        opts = { verify_ssl: OpenSSL::SSL::VERIFY_PEER }
+        opts = { verify_ssl: OpenSSL::SSL::VERIFY_NONE }
 
         if ca_pem.present?
           opts[:cert_store] = OpenSSL::X509::Store.new
diff --git a/app/models/concerns/prometheus_adapter.rb b/app/models/concerns/prometheus_adapter.rb
new file mode 100644
index 00000000000..fea99a7cd35
--- /dev/null
+++ b/app/models/concerns/prometheus_adapter.rb
@@ -0,0 +1,48 @@
+module PrometheusAdapter
+  extend ActiveSupport::Concern
+
+  included do
+    include ReactiveCaching
+
+    self.reactive_cache_key = ->(adapter) { [adapter.class.model_name.singular, adapter.id] }
+    self.reactive_cache_lease_timeout = 30.seconds
+    self.reactive_cache_refresh_interval = 30.seconds
+    self.reactive_cache_lifetime = 1.minute
+
+    def prometheus_client
+      raise NotImplementedError
+    end
+
+    def prometheus_client_wrapper
+      Gitlab::PrometheusClient.new(prometheus_client)
+    end
+
+    def can_query?
+      prometheus_client.present?
+    end
+
+    def query(query_name, *args)
+      return unless can_query?
+
+      query_class = Gitlab::Prometheus::Queries.const_get("#{query_name.to_s.classify}Query")
+
+      args.map! { |arg| arg.id }
+
+      with_reactive_cache(query_class.name, *args, &query_class.method(:transform_reactive_result))
+    end
+
+    # Cache metrics for specific environment
+    def calculate_reactive_cache(query_class_name, *args)
+      return unless prometheus_client
+
+      data = Kernel.const_get(query_class_name).new(prometheus_client_wrapper).query(*args)
+      {
+        success: true,
+        data: data,
+        last_update: Time.now.utc
+      }
+    rescue Gitlab::PrometheusClient::Error => err
+      { success: false, result: err.message }
+    end
+  end
+end
diff --git a/app/models/deployment.rb b/app/models/deployment.rb
index b6cf168d60e..66e61c06765 100644
--- a/app/models/deployment.rb
+++ b/app/models/deployment.rb
@@ -98,28 +98,29 @@ class Deployment < ActiveRecord::Base
   end
 
   def has_metrics?
-    project.monitoring_service.present?
+    prometheus_adapter&.can_query?
   end
 
   def metrics
     return {} unless has_metrics?
 
-    project.monitoring_service.deployment_metrics(self)
-  end
-
-  def has_additional_metrics?
-    project.prometheus_service.present?
+    metrics = prometheus_adapter.query(:deployment, self)
+    metrics&.merge(deployment_time: created_at.to_i) || {}
   end
 
   def additional_metrics
-    return {} unless project.prometheus_service.present?
+    return {} unless has_metrics?
 
-    metrics = project.prometheus_service.additional_deployment_metrics(self)
+    metrics = prometheus_adapter.query(:additional_metrics_deployment, self)
     metrics&.merge(deployment_time: created_at.to_i) || {}
   end
 
   private
 
+  def prometheus_adapter
+    environment.prometheus_adapter
+  end
+
   def ref_path
     File.join(environment.ref_path, 'deployments', iid.to_s)
   end
diff --git a/app/models/environment.rb b/app/models/environment.rb
index f78c21aebe5..2a91bd07b28 100644
--- a/app/models/environment.rb
+++ b/app/models/environment.rb
@@ -146,21 +146,15 @@ class Environment < ActiveRecord::Base
   end
 
   def has_metrics?
-    project.monitoring_service.present? && available? && last_deployment.present?
+    prometheus_adapter&.can_query? && available? && last_deployment.present?
   end
 
   def metrics
-    project.monitoring_service.environment_metrics(self) if has_metrics?
-  end
-
-  def has_additional_metrics?
-    project.prometheus_service.present? && available? && last_deployment.present?
+    prometheus_adapter.query(:environment, self) if has_metrics?
   end
 
   def additional_metrics
-    if has_additional_metrics?
-      project.prometheus_service.additional_environment_metrics(self)
-    end
+    prometheus_adapter.query(:additional_metrics_environment, self) if has_metrics?
   end
 
   def slug
@@ -226,6 +220,27 @@ class Environment < ActiveRecord::Base
     self.environment_type || self.name
   end
 
+  def prometheus_adapter
+    @prometheus_adapter ||= if service_prometheus_adapter.can_query?
+                              service_prometheus_adapter
+                            else
+                              cluster_prometheus_adapter
+                            end
+  end
+
+  def service_prometheus_adapter
+    project.find_or_initialize_service('prometheus')
+  end
+
+  def cluster_prometheus_adapter
+    # sort results by descending order based on environment_scope being longer
+    # thus more closely matching environment slug
+    clusters = project.clusters.enabled.for_environment(self).sort_by { |c| c.environment_scope&.length }.reverse!
+
+    cluster = clusters&.detect { |cluster| cluster.application_prometheus&.installed? }
+    cluster&.application_prometheus
+  end
+
   private
 
   # Slugifying a name may remove the uniqueness guarantee afforded by it being
diff --git a/app/models/project_services/monitoring_service.rb b/app/models/project_services/monitoring_service.rb
index ee9cd78327a..9af68b4e821 100644
--- a/app/models/project_services/monitoring_service.rb
+++ b/app/models/project_services/monitoring_service.rb
@@ -9,11 +9,11 @@ class MonitoringService < Service
     %w()
   end
 
-  def environment_metrics(environment)
+  def can_query?
     raise NotImplementedError
   end
 
-  def deployment_metrics(deployment)
+  def query(_, *_)
     raise NotImplementedError
   end
 end
diff --git a/app/models/project_services/prometheus_service.rb b/app/models/project_services/prometheus_service.rb
index 58731451429..7a0c9a33e70 100644
--- a/app/models/project_services/prometheus_service.rb
+++ b/app/models/project_services/prometheus_service.rb
@@ -1,9 +1,5 @@
 class PrometheusService < MonitoringService
-  include ReactiveService
-
-  self.reactive_cache_lease_timeout = 30.seconds
-  self.reactive_cache_refresh_interval = 30.seconds
-  self.reactive_cache_lifetime = 1.minute
+  include PrometheusAdapter
 
   #  Access to prometheus is directly through the API
   prop_accessor :api_url
@@ -66,63 +62,15 @@ class PrometheusService < MonitoringService
 
   # Check we can connect to the Prometheus API
   def test(*args)
-    client.ping
+    Gitlab::PrometheusClient.new(prometheus_client).ping
 
     { success: true, result: 'Checked API endpoint' }
   rescue Gitlab::PrometheusClient::Error => err
     { success: false, result: err }
   end
 
-  def environment_metrics(environment)
-    with_reactive_cache(Gitlab::Prometheus::Queries::EnvironmentQuery.name, environment.id, &rename_field(:data, :metrics))
-  end
-
-  def deployment_metrics(deployment)
-    metrics = with_reactive_cache(Gitlab::Prometheus::Queries::DeploymentQuery.name, deployment.environment.id, deployment.id, &rename_field(:data, :metrics))
-    metrics&.merge(deployment_time: deployment.created_at.to_i) || {}
-  end
-
-  def additional_environment_metrics(environment)
-    with_reactive_cache(Gitlab::Prometheus::Queries::AdditionalMetricsEnvironmentQuery.name, environment.id, &:itself)
-  end
-
-  def additional_deployment_metrics(deployment)
-    with_reactive_cache(Gitlab::Prometheus::Queries::AdditionalMetricsDeploymentQuery.name, deployment.environment.id, deployment.id, &:itself)
-  end
-
-  def matched_metrics
-    with_reactive_cache(Gitlab::Prometheus::Queries::MatchedMetricsQuery.name, &:itself)
-  end
-
-  # Cache metrics for specific environment
-  def calculate_reactive_cache(query_class_name, *args)
-    return unless active? && project && !project.pending_delete?
-
-    environment_id = args.first
-    client = client(environment_id)
-
-    data = Kernel.const_get(query_class_name).new(client).query(*args)
-    {
-      success: true,
-      data: data,
-      last_update: Time.now.utc
-    }
-  rescue Gitlab::PrometheusClient::Error => err
-    { success: false, result: err.message }
-  end
-
-  def client(environment_id = nil)
-    if manual_configuration?
-      Gitlab::PrometheusClient.new(RestClient::Resource.new(api_url))
-    else
-      cluster = cluster_with_prometheus(environment_id)
-      raise Gitlab::PrometheusClient::Error, "couldn't find cluster with Prometheus installed" unless cluster
-
-      rest_client = client_from_cluster(cluster)
-      raise Gitlab::PrometheusClient::Error, "couldn't create proxy Prometheus client" unless rest_client
-
-      Gitlab::PrometheusClient.new(rest_client)
-    end
+  def prometheus_client
+    RestClient::Resource.new(api_url) if api_url && manual_configuration? && active?
   end
 
   def prometheus_installed?
@@ -134,31 +82,6 @@ class PrometheusService < MonitoringService
 
   private
 
-  def cluster_with_prometheus(environment_id = nil)
-    clusters = if environment_id
-                 ::Environment.find_by(id: environment_id).try do |env|
-                   # sort results by descending order based on environment_scope being longer
-                   # thus more closely matching environment slug
-                   project.clusters.enabled.for_environment(env).sort_by { |c| c.environment_scope&.length }.reverse!
-                 end
-               else
-                 project.clusters.enabled.for_all_environments
-               end
-
-    clusters&.detect { |cluster| cluster.application_prometheus&.installed? }
-  end
-
-  def client_from_cluster(cluster)
-    cluster.application_prometheus.proxy_client
-  end
-
-  def rename_field(old_field, new_field)
-    -> (metrics) do
-      metrics[new_field] = metrics.delete(old_field)
-      metrics
-    end
-  end
-
   def synchronize_service_state!
     self.active = prometheus_installed? || manual_configuration?