Adding --check-defaults to validator

When troubleshooting, it's often interesting to see the deltas between
the default or sample configs and the running configuration. The
oslo_config.validator is a great tool to integrate this feature across
all projects. It can also be easily captured by data collection tools
like sosreports with the current deployment packages.

Change-Id: I172d82f19a81282093b0d5f7ae4c1817801cd887
Signed-off-by: David Vallee Delisle <dvd@redhat.com>

4 files changed, 302 insertions, 9 deletions

diff --git a/doc/source/cli/validator.rst b/doc/source/cli/validator.rst
index d829b8d..23ae424 100644
--- a/doc/source/cli/validator.rst
+++ b/doc/source/cli/validator.rst
@@ -32,7 +32,7 @@ Here's an example of using the validator on Nova as installed by Devstack (with
 the option [foo]/bar added to demonstrate a failure)::
 
     $ oslo-config-validator --config-file /opt/stack/nova/etc/nova/nova-config-generator.conf --input-file /etc/nova/nova.conf
-    ERROR:root:foo/bar not found
+    ERROR:root:foo/bar is not part of the sample config
     INFO:root:Ignoring missing option "project_domain_name" from group "keystone_authtoken" because the group is known to have incomplete sample config data and thus cannot be validated properly.
     INFO:root:Ignoring missing option "project_name" from group "keystone_authtoken" because the group is known to have incomplete sample config data and thus cannot be validated properly.
     INFO:root:Ignoring missing option "user_domain_name" from group "keystone_authtoken" because the group is known to have incomplete sample config data and thus cannot be validated properly.
@@ -58,7 +58,7 @@ a sample config file ``config-data.yaml`` created by the config generator (with
 the option [foo]/bar added to demonstrate a failure)::
 
     $ oslo-config-validator --opt-data config-data.yaml --input-file /etc/nova/nova.conf
-    ERROR:root:foo/bar not found
+    ERROR:root:foo/bar is not part of the sample config
     INFO:root:Ignoring missing option "project_domain_name" from group "keystone_authtoken" because the group is known to have incomplete sample config data and thus cannot be validated properly.
     INFO:root:Ignoring missing option "project_name" from group "keystone_authtoken" because the group is known to have incomplete sample config data and thus cannot be validated properly.
     INFO:root:Ignoring missing option "user_domain_name" from group "keystone_authtoken" because the group is known to have incomplete sample config data and thus cannot be validated properly.
@@ -66,6 +66,158 @@ the option [foo]/bar added to demonstrate a failure)::
     INFO:root:Ignoring missing option "username" from group "keystone_authtoken" because the group is known to have incomplete sample config data and thus cannot be validated properly.
     INFO:root:Ignoring missing option "auth_url" from group "keystone_authtoken" because the group is known to have incomplete sample config data and thus cannot be validated properly.
 
+Comparing configuration with the default configuration
+------------------------------------------------------
+
+.. note:: For most accurate results, the validation should be done using the
+          same version of the code as the system whose config file is being
+          validated.
+
+Comparing the default configuration with the current configuration can help
+operators with troubleshooting issues. Since the generator config is not always
+available in production environment, we can pass the ``--namespace`` arguments.
+In addition to the ``--namespace``, we need to pass a ``--input-file`` as well
+as the ``--check-defaults``.
+
+Some options are ignored by default but this behavior can be overridden with
+the ``--exclude-options`` list argument.
+
+Here's an example of using the validator on Nova::
+
+    $ oslo-config-validator --input-file /etc/nova/nova.conf \
+                            --check-defaults \
+                            --namespace nova.conf \
+                            --namespace oslo.log \
+                            --namespace oslo.messaging \
+                            --namespace oslo.policy \
+                            --namespace oslo.privsep \
+                            --namespace oslo.service.periodic_task \
+                            --namespace oslo.service.service \
+                            --namespace oslo.db \
+                            --namespace oslo.db.concurrency \
+                            --namespace oslo.middleware \
+                            --namespace oslo.concurrency \
+                            --namespace keystonemiddleware.auth_token \
+                            --namespace osprofiler
+    INFO:keyring.backend:Loading Gnome
+    INFO:keyring.backend:Loading Google
+    INFO:keyring.backend:Loading Windows (alt)
+    INFO:keyring.backend:Loading file
+    INFO:keyring.backend:Loading keyczar
+    INFO:keyring.backend:Loading multi
+    INFO:keyring.backend:Loading pyfs
+    WARNING:root:DEFAULT/compute_driver sample value is empty but input-file has libvirt.LibvirtDriver
+    WARNING:root:DEFAULT/allow_resize_to_same_host sample value is empty but input-file has True
+    WARNING:root:DEFAULT/default_ephemeral_format sample value is empty but input-file has ext4
+    WARNING:root:DEFAULT/pointer_model sample value ['usbtablet'] is not in ['ps2mouse']
+    WARNING:root:DEFAULT/instances_path sample value ['$state_path/instances'] is not in ['/opt/stack/data/nova/instances']
+    WARNING:root:DEFAULT/shutdown_timeout sample value ['60'] is not in ['0']
+    INFO:root:DEFAULT/my_ip Ignoring option because it is part of the excluded patterns. This can be changed with the --exclude-options argument.
+    WARNING:root:DEFAULT/state_path sample value ['$pybasedir'] is not in ['/opt/stack/data/nova']
+    INFO:root:DEFAULT/osapi_compute_listen Ignoring option because it is part of the excluded patterns. This can be changed with the --exclude-options argument.
+    WARNING:root:DEFAULT/osapi_compute_workers sample value is empty but input-file has 2
+    WARNING:root:DEFAULT/metadata_workers sample value is empty but input-file has 2
+    WARNING:root:DEFAULT/graceful_shutdown_timeout sample value ['60'] is not in ['5']
+    INFO:root:DEFAULT/transport_url Ignoring option because it is part of the excluded patterns. This can be changed with the --exclude-options argument.
+    WARNING:root:DEFAULT/debug sample value is empty but input-file has True
+    WARNING:root:DEFAULT/logging_context_format_string sample value ['%(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [%(request_id)s %(user_identity)s] %(instance)s%(message)s'] is not in ['%(color)s%(levelname)s %(name)s [\x1b[01;36m%(global_request_id)s %(request_id)s \x1b[00;36m%(project_name)s %(user_name)s%(color)s] \x1b[01;35m%(instance)s%(color)s%(message)s\x1b[00m']
+    WARNING:root:DEFAULT/logging_default_format_string sample value ['%(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [-] %(instance)s%(message)s'] is not in ['%(color)s%(levelname)s %(name)s [\x1b[00;36m-%(color)s] \x1b[01;35m%(instance)s%(color)s%(message)s\x1b[00m']
+    WARNING:root:DEFAULT/logging_debug_format_suffix sample value ['%(funcName)s %(pathname)s:%(lineno)d'] is not in ['\x1b[00;33m{{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}}\x1b[00m']
+    WARNING:root:DEFAULT/logging_exception_prefix sample value ['%(asctime)s.%(msecs)03d %(process)d ERROR %(name)s %(instance)s'] is not in ['ERROR %(name)s \x1b[01;35m%(instance)s\x1b[00m']
+    WARNING:root:Group api from the sample config is not defined in input-file
+    WARNING:root:cache/backend sample value ['dogpile.cache.null'] is not in ['dogpile.cache.memcached']
+    WARNING:root:cache/enabled sample value is empty but input-file has True
+    WARNING:root:cinder/os_region_name sample value is empty but input-file has RegionOne
+    WARNING:root:cinder/auth_type sample value is empty but input-file has password
+    INFO:root:cinder/auth_url Ignoring option because it is part of the excluded patterns. This can be changed with the --exclude-options argument.
+    WARNING:root:cinder/project_name sample value is empty but input-file has service
+    WARNING:root:cinder/project_domain_name sample value is empty but input-file has Default
+    INFO:root:cinder/username Ignoring option because it is part of the excluded patterns. This can be changed with the --exclude-options argument.
+    WARNING:root:cinder/user_domain_name sample value is empty but input-file has Default
+    INFO:root:cinder/password Ignoring option because it is part of the excluded patterns. This can be changed with the --exclude-options argument.
+    WARNING:root:Group compute from the sample config is not defined in input-file
+    WARNING:root:conductor/workers sample value is empty but input-file has 2
+    WARNING:root:Group console from the sample config is not defined in input-file
+    WARNING:root:Group consoleauth from the sample config is not defined in input-file
+    WARNING:root:Group cyborg from the sample config is not defined in input-file
+    INFO:root:api_database/connection Ignoring option because it is part of the excluded patterns. This can be changed with the --exclude-options argument.
+    WARNING:root:Group devices from the sample config is not defined in input-file
+    WARNING:root:Group ephemeral_storage_encryption from the sample config is not defined in input-file
+    WARNING:root:Group glance from the sample config is not defined in input-file
+    WARNING:root:Group guestfs from the sample config is not defined in input-file
+    WARNING:root:Group hyperv from the sample config is not defined in input-file
+    WARNING:root:Group image_cache from the sample config is not defined in input-file
+    WARNING:root:Group ironic from the sample config is not defined in input-file
+    WARNING:root:key_manager/fixed_key sample value is empty but input-file has bae3516cc1c0eb18b05440eba8012a4a880a2ee04d584a9c1579445e675b12defdc716ec
+    WARNING:root:key_manager/backend sample value ['barbican'] is not in ['nova.keymgr.conf_key_mgr.ConfKeyManager']
+    WARNING:root:Group barbican from the sample config is not defined in input-file
+    WARNING:root:Group vault from the sample config is not defined in input-file
+    WARNING:root:Group keystone from the sample config is not defined in input-file
+    INFO:root:libvirt/live_migration_uri Ignoring option because it is part of the excluded patterns. This can be changed with the --exclude-options argument.
+    WARNING:root:libvirt/cpu_mode sample value is empty but input-file has none
+    WARNING:root:Group mks from the sample config is not defined in input-file
+    WARNING:root:neutron/default_floating_pool sample value ['nova'] is not in ['public']
+    WARNING:root:neutron/service_metadata_proxy sample value is empty but input-file has True
+    WARNING:root:neutron/auth_type sample value is empty but input-file has password
+    INFO:root:neutron/auth_url Ignoring option because it is part of the excluded patterns. This can be changed with the --exclude-options argument.
+    WARNING:root:neutron/project_name sample value is empty but input-file has service
+    WARNING:root:neutron/project_domain_name sample value is empty but input-file has Default
+    INFO:root:neutron/username Ignoring option because it is part of the excluded patterns. This can be changed with the --exclude-options argument.
+    WARNING:root:neutron/user_domain_name sample value is empty but input-file has Default
+    INFO:root:neutron/password Ignoring option because it is part of the excluded patterns. This can be changed with the --exclude-options argument.
+    WARNING:root:neutron/region_name sample value is empty but input-file has RegionOne
+    WARNING:root:Group pci from the sample config is not defined in input-file
+    WARNING:root:placement/auth_type sample value is empty but input-file has password
+    INFO:root:placement/auth_url Ignoring option because it is part of the excluded patterns. This can be changed with the --exclude-options argument.
+    WARNING:root:placement/project_name sample value is empty but input-file has service
+    WARNING:root:placement/project_domain_name sample value is empty but input-file has Default
+    INFO:root:placement/username Ignoring option because it is part of the excluded patterns. This can be changed with the --exclude-options argument.
+    WARNING:root:placement/user_domain_name sample value is empty but input-file has Default
+    INFO:root:placement/password Ignoring option because it is part of the excluded patterns. This can be changed with the --exclude-options argument.
+    WARNING:root:placement/region_name sample value is empty but input-file has RegionOne
+    WARNING:root:Group powervm from the sample config is not defined in input-file
+    WARNING:root:Group quota from the sample config is not defined in input-file
+    WARNING:root:Group rdp from the sample config is not defined in input-file
+    WARNING:root:Group remote_debug from the sample config is not defined in input-file
+    WARNING:root:scheduler/workers sample value is empty but input-file has 2
+    WARNING:root:filter_scheduler/track_instance_changes sample value ['True'] is not in ['False']
+    WARNING:root:filter_scheduler/enabled_filters sample value ['AvailabilityZoneFilter', 'ComputeFilter', 'ComputeCapabilitiesFilter', 'ImagePropertiesFilter', 'ServerGroupAntiAffinityFilter', 'ServerGroupAffinityFilter'] is not in ['AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,SameHostFilter,DifferentHostFilter']
+    WARNING:root:Group metrics from the sample config is not defined in input-file
+    WARNING:root:Group serial_console from the sample config is not defined in input-file
+    WARNING:root:Group service_user from the sample config is not defined in input-file
+    WARNING:root:Group spice from the sample config is not defined in input-file
+    WARNING:root:upgrade_levels/compute sample value is empty but input-file has auto
+    WARNING:root:Group vendordata_dynamic_auth from the sample config is not defined in input-file
+    WARNING:root:Group vmware from the sample config is not defined in input-file
+    WARNING:root:Group vnc from the sample config is not defined in input-file
+    WARNING:root:Group workarounds from the sample config is not defined in input-file
+    WARNING:root:wsgi/api_paste_config sample value ['api-paste.ini'] is not in ['/etc/nova/api-paste.ini']
+    WARNING:root:Group zvm from the sample config is not defined in input-file
+    WARNING:root:oslo_concurrency/lock_path sample value is empty but input-file has /opt/stack/data/nova
+    WARNING:root:Group oslo_middleware from the sample config is not defined in input-file
+    WARNING:root:Group cors from the sample config is not defined in input-file
+    WARNING:root:Group healthcheck from the sample config is not defined in input-file
+    WARNING:root:Group oslo_messaging_amqp from the sample config is not defined in input-file
+    WARNING:root:oslo_messaging_notifications/driver sample value is empty but input-file has messagingv2
+    INFO:root:oslo_messaging_notifications/transport_url Ignoring option because it is part of the excluded patterns. This can be changed with the --exclude-options argument.
+    WARNING:root:Group oslo_messaging_rabbit from the sample config is not defined in input-file
+    WARNING:root:Group oslo_messaging_kafka from the sample config is not defined in input-file
+    WARNING:root:Group oslo_policy from the sample config is not defined in input-file
+    WARNING:root:Group privsep from the sample config is not defined in input-file
+    WARNING:root:Group profiler from the sample config is not defined in input-file
+    INFO:root:database/connection Ignoring option because it is part of the excluded patterns. This can be changed with the --exclude-options argument.
+    WARNING:root:keystone_authtoken/interface sample value ['internal'] is not in ['public']
+    WARNING:root:keystone_authtoken/cafile sample value is empty but input-file has /opt/stack/data/ca-bundle.pem
+    WARNING:root:keystone_authtoken/memcached_servers sample value is empty but input-file has localhost:11211
+    WARNING:root:keystone_authtoken/auth_type sample value is empty but input-file has password
+    ERROR:root:neutron/auth_strategy is not part of the sample config
+    INFO:root:Ignoring missing option "project_domain_name" from group "keystone_authtoken" because the group is known to have incomplete sample config data and thus cannot be validated properly.
+    INFO:root:Ignoring missing option "project_name" from group "keystone_authtoken" because the group is known to have incomplete sample config data and thus cannot be validated properly.
+    INFO:root:Ignoring missing option "user_domain_name" from group "keystone_authtoken" because the group is known to have incomplete sample config data and thus cannot be validated properly.
+    INFO:root:Ignoring missing option "password" from group "keystone_authtoken" because the group is known to have incomplete sample config data and thus cannot be validated properly.
+    INFO:root:Ignoring missing option "username" from group "keystone_authtoken" because the group is known to have incomplete sample config data and thus cannot be validated properly.
+    INFO:root:Ignoring missing option "auth_url" from group "keystone_authtoken" because the group is known to have incomplete sample config data and thus cannot be validated properly
+
 Handling Dynamic Groups
 -----------------------
 
diff --git a/oslo_config/tests/test_validator.py b/oslo_config/tests/test_validator.py
index de912ac..383340f 100644
--- a/oslo_config/tests/test_validator.py
+++ b/oslo_config/tests/test_validator.py
@@ -21,13 +21,51 @@ from oslo_config import fixture
 from oslo_config import validator
 
 
-OPT_DATA = {'options': {'foo': {'opts': [{'name': 'opt'}]},
-                        'bar': {'opts': [{'name': 'opt'}]},
-                        },
-            'deprecated_options': {'bar': [{'name': 'opt'}]}}
+OPT_DATA = {
+  "options": {
+    "foo": {
+      "opts": [
+        {
+          "name": "opt",
+          "default": 1
+        }
+      ]
+    },
+    "bar": {
+      "opts": [
+        {
+          "name": "opt",
+          "default": 2
+        },
+        {
+          "name": "foo-bar",
+          "dest": "foo_bar",
+          "default": 2
+        },
+        {
+          "name": "bar-foo",
+          "dest": "bar_foo",
+          "default": 2
+        }
+      ]
+    }
+  },
+  "deprecated_options": {
+    "bar": [
+      {
+        "name": "opt",
+        "default": 3
+      }
+    ]
+  }
+}
 VALID_CONF = """
 [foo]
-opt = value
+opt = 1
+[bar]
+opt = 3
+foo-bar = 3
+bar_foo = 3
 """
 DEPRECATED_CONF = """
 [bar]
@@ -106,5 +144,15 @@ class TestValidator(base.BaseTestCase):
         with mock.patch('builtins.open', m):
             self.assertEqual(0, validator._validate(self.conf))
 
+    @mock.patch('oslo_config.validator.load_opt_data')
+    def test_check_defaults(self, mock_lod):
+        mock_lod.return_value = OPT_DATA
+        self.conf_fixture.config(opt_data='mocked.yaml',
+                                 input_file='mocked.conf',
+                                 check_defaults=True)
+        m = mock.mock_open(read_data=VALID_CONF)
+        with mock.patch('builtins.open', m):
+            self.assertEqual(0, validator._validate(self.conf))
+
     def test_invalid_options(self):
         self.assertRaises(RuntimeError, validator._validate, self.conf)
diff --git a/oslo_config/validator.py b/oslo_config/validator.py
index 725a868..938ee65 100644
--- a/oslo_config/validator.py
+++ b/oslo_config/validator.py
@@ -22,6 +22,7 @@ project then it returns those errors.
 """
 
 import logging
+import re
 import sys
 
 try:
@@ -36,6 +37,11 @@ import yaml
 from oslo_config import cfg
 from oslo_config import generator
 
+VALIDATE_DEFAULTS_EXCLUSIONS = [
+    '.*_ur(i|l)', '.*connection', 'password', 'username', 'my_ip',
+    'host(name)?', 'glance_api_servers', 'osapi_volume_listen',
+    'osapi_compute_listen',
+]
 
 _validator_opts = [
     cfg.MultiStrOpt(
@@ -51,6 +57,16 @@ _validator_opts = [
         help='Path to a YAML file containing definitions of options, as '
              'output by the config generator.'),
     cfg.BoolOpt(
+        'check-defaults',
+        default=False,
+        help='Report differences between the sample values and current '
+             'values.'),
+    cfg.ListOpt(
+        'exclude-options',
+        default=VALIDATE_DEFAULTS_EXCLUSIONS,
+        help='Exclude options matching these patterns when comparing '
+             'the current and sample configurations.'),
+    cfg.BoolOpt(
         'fatal-warnings',
         default=False,
         help='Report failure if any warnings are found.'),
@@ -86,6 +102,73 @@ def _validate_deprecated_opt(group, option, opt_data):
     return option in name_data
 
 
+def _validate_defaults(sections, opt_data, conf):
+    """Compares the current and sample configuration and reports differences
+
+    :param section: ConfigParser instance
+    :param opt_data: machine readable data from the generator instance
+    :param conf: ConfigOpts instance
+    :returns: boolean wether or not warnings were reported
+    """
+    warnings = False
+    # Generating regex objects from ListOpt
+    exclusion_regexes = []
+    for pattern in conf.exclude_options:
+        exclusion_regexes.append(re.compile(pattern))
+    for group, opts in opt_data['options'].items():
+        if group in conf.exclude_group:
+            continue
+        if group not in sections:
+            logging.warning(
+                'Group %s from the sample config is not defined in '
+                'input-file', group)
+            continue
+        for opt in opts['opts']:
+            # We need to convert the defaults into a list to find
+            # intersections. defaults are only a list if they can
+            # be defined multiple times, but configparser only
+            # returns list
+            if not isinstance(opt['default'], list):
+                defaults = [str(opt['default'])]
+            else:
+                defaults = opt['default']
+
+            # Apparently, there's multiple naming conventions for
+            # options, 'name' is mostly with hyphens, and 'dest'
+            # is represented with underscores.
+            opt_names = set([opt['name'], opt.get('dest')])
+            if not opt_names.intersection(sections[group]):
+                continue
+            try:
+                value = sections[group][opt['name']]
+                keyname = opt['name']
+            except KeyError:
+                value = sections[group][opt.get('dest')]
+                keyname = opt.get('dest')
+
+            if any(rex.fullmatch(keyname) for rex in exclusion_regexes):
+                logging.info(
+                    '%s/%s Ignoring option because it is part of the excluded '
+                    'patterns. This can be changed with the --exclude-options '
+                    'argument', group, keyname)
+                continue
+
+            if len(value) > 1:
+                logging.info(
+                    '%s/%s defined %s times', group, keyname, len(value))
+            if not opt['default']:
+                logging.warning(
+                    '%s/%s sample value is empty but input-file has %s',
+                    group, keyname, ", ".join(value))
+                warnings = True
+            elif not frozenset(defaults).intersection(value):
+                logging.warning(
+                    '%s/%s sample value %s is not in %s',
+                    group, keyname, defaults, value)
+                warnings = True
+    return warnings
+
+
 def _validate_opt(group, option, opt_data):
     if group not in opt_data['options']:
         return False
@@ -114,12 +197,14 @@ def _validate(conf):
     parser.parse()
     warnings = False
     errors = False
+    if conf.check_defaults:
+        warnings = _validate_defaults(sections, opt_data, conf)
     for section, options in sections.items():
         if section in conf.exclude_group:
             continue
         for option in options:
             if _validate_deprecated_opt(section, option, opt_data):
-                logging.warn('Deprecated opt %s/%s found', section, option)
+                logging.warning('Deprecated opt %s/%s found', section, option)
                 warnings = True
             elif not _validate_opt(section, option, opt_data):
                 if section in KNOWN_BAD_GROUPS:
@@ -129,7 +214,8 @@ def _validate(conf):
                                  'cannot be validated properly.',
                                  option, section)
                     continue
-                logging.error('%s/%s not found', section, option)
+                logging.error('%s/%s is not part of the sample config',
+                              section, option)
                 errors = True
     if errors or (warnings and conf.fatal_warnings):
         return 1
diff --git a/releasenotes/notes/validator-check-defaults-e7b596a2fde781a8.yaml b/releasenotes/notes/validator-check-defaults-e7b596a2fde781a8.yaml
new file mode 100644
index 0000000..3fdbfe6
--- /dev/null
+++ b/releasenotes/notes/validator-check-defaults-e7b596a2fde781a8.yaml
@@ -0,0 +1,7 @@
+---
+features:
+  - |
+    Add a ``--check-defaults`` flag to ``oslo-config-validator``. When set,
+    ``oslo-config-validator`` will compare the ``input-file`` with the sample
+    generated configuration and flag any discrepancies. It also obeys the
+    standard ``--exclude-group`` and ``--fatal-warning`` options.