[goal] Deprecate the JSON formatted policy file

As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:

1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.

2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.

Also replace policy.json to policy.yaml ref from doc and tests.

[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html

Change-Id: Ic65d2fd6ce7215b4a47a6fb41b9cbf991f27773b

13 files changed, 68 insertions, 21 deletions

diff --git a/doc/source/admin/cli-manage-projects-users-and-roles.rst b/doc/source/admin/cli-manage-projects-users-and-roles.rst
index f27979d2c..8d2f83702 100644
--- a/doc/source/admin/cli-manage-projects-users-and-roles.rst
+++ b/doc/source/admin/cli-manage-projects-users-and-roles.rst
@@ -10,8 +10,8 @@ define which actions users can perform. You assign roles to
 user-project pairs.
 
 You can define actions for OpenStack service roles in the
-``/etc/PROJECT/policy.json`` files. For example, define actions for
-Compute service roles in the ``/etc/nova/policy.json`` file.
+``/etc/PROJECT/policy.yaml`` files. For example, define actions for
+Compute service roles in the ``/etc/nova/policy.yaml`` file.
 
 You can manage projects, users, and roles independently from each other.
 
diff --git a/doc/source/admin/identity-concepts.rst b/doc/source/admin/identity-concepts.rst
index 3d615c0da..0f8cfc51f 100644
--- a/doc/source/admin/identity-concepts.rst
+++ b/doc/source/admin/identity-concepts.rst
@@ -122,9 +122,9 @@ Identity user management examples:
      Individual services assign meaning to roles, typically through
      limiting or granting access to users with the role to the
      operations that the service supports. Role access is typically
-     configured in the service's ``policy.json`` file. For example,
+     configured in the service's ``policy.yaml`` file. For example,
      to limit Compute access to the ``compute-user`` role, edit the
-     Compute service's ``policy.json`` file to require this role for
+     Compute service's ``policy.yaml`` file to require this role for
      Compute operations.
 
 The Identity service assigns a project and a role to a user. You might
@@ -139,25 +139,25 @@ A user can have different roles in different projects. For example, Alice
 might also have the ``admin`` role in the ``Cyberdyne`` project. A user
 can also have multiple roles in the same project.
 
-The ``/etc/[SERVICE_CODENAME]/policy.json`` file controls the
+The ``/etc/[SERVICE_CODENAME]/policy.yaml`` file controls the
 tasks that users can perform for a given service. For example, the
-``/etc/nova/policy.json`` file specifies the access policy for the
-Compute service, the ``/etc/glance/policy.json`` file specifies
+``/etc/nova/policy.yaml`` file specifies the access policy for the
+Compute service, the ``/etc/glance/policy.yaml`` file specifies
 the access policy for the Image service, and the
-``/etc/keystone/policy.json`` file specifies the access policy for
+``/etc/keystone/policy.yaml`` file specifies the access policy for
 the Identity service.
 
-The default ``policy.json`` files in the Compute, Identity, and
+The default ``policy.yaml`` files in the Compute, Identity, and
 Image services recognize only the ``admin`` role. Any user with
 any role in a project can access all operations that do not require the
 ``admin`` role.
 
 To restrict users from performing operations in, for example, the
 Compute service, you must create a role in the Identity service and
-then modify the ``/etc/nova/policy.json`` file so that this role
+then modify the ``/etc/nova/policy.yaml`` file so that this role
 is required for Compute operations.
 
-For example, the following line in the ``/etc/cinder/policy.json``
+For example, the following line in the ``/etc/cinder/policy.yaml``
 file does not restrict which users can create volumes:
 
 .. code-block:: none
diff --git a/doc/source/admin/upgrading.rst b/doc/source/admin/upgrading.rst
index 687dba4e3..709d98dac 100644
--- a/doc/source/admin/upgrading.rst
+++ b/doc/source/admin/upgrading.rst
@@ -42,7 +42,7 @@ Plan your upgrade:
   to re-read the release notes for the previous release (or two!).
 
 * Prepare your new configuration files, including ``keystone.conf``,
-  ``logging.conf``, ``policy.json``, ``keystone-paste.ini``, and anything else
+  ``logging.conf``, ``policy.yaml``, ``keystone-paste.ini``, and anything else
   in ``/etc/keystone/``, by customizing the corresponding files from the next
   release.
 
diff --git a/doc/source/configuration/policy.rst b/doc/source/configuration/policy.rst
index daafdea51..3f80c5ee2 100644
--- a/doc/source/configuration/policy.rst
+++ b/doc/source/configuration/policy.rst
@@ -2,6 +2,15 @@
 Policy configuration
 ====================
 
+.. warning::
+
+   JSON formatted policy file is deprecated since Keystone 19.0.0 (Wallaby).
+   This `oslopolicy-convert-json-to-yaml`__ tool will migrate your existing
+   JSON-formatted policy file to YAML in a backward-compatible way.
+
+.. __: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html
+
+
 Configuration
 ~~~~~~~~~~~~~
 
diff --git a/doc/source/getting-started/policy_mapping.rst b/doc/source/getting-started/policy_mapping.rst
index 2975b45b3..a7cb27cfa 100644
--- a/doc/source/getting-started/policy_mapping.rst
+++ b/doc/source/getting-started/policy_mapping.rst
@@ -2,7 +2,7 @@
 Mapping of policy target to API
 ===============================
 
-The following table shows the target in the policy.json file for each API.
+The following table shows the target in the policy.yaml file for each API.
 
 =========================================================  ===
 Target                                                     API
diff --git a/keystone/cmd/status.py b/keystone/cmd/status.py
index 3585c2e2b..64b2e62f3 100644
--- a/keystone/cmd/status.py
+++ b/keystone/cmd/status.py
@@ -12,6 +12,7 @@
 
 from oslo_policy import _checks
 from oslo_policy import policy
+from oslo_upgradecheck import common_checks
 from oslo_upgradecheck import upgradecheck
 
 from keystone.common import driver_hints
@@ -86,6 +87,8 @@ class Checks(upgradecheck.UpgradeCommands):
          check_trust_policies_are_not_empty),
         ("Check default roles are immutable",
          check_default_roles_are_immutable),
+        ("Policy File JSON to YAML Migration",
+         (common_checks.check_policy_json, {'conf': CONF})),
     )
 
 
diff --git a/keystone/common/rbac_enforcer/enforcer.py b/keystone/common/rbac_enforcer/enforcer.py
index ca6a8e7b2..7add048ce 100644
--- a/keystone/common/rbac_enforcer/enforcer.py
+++ b/keystone/common/rbac_enforcer/enforcer.py
@@ -14,6 +14,7 @@ import functools
 
 import flask
 from oslo_log import log
+from oslo_policy import opts
 from oslo_policy import policy as common_policy
 from oslo_utils import strutils
 
@@ -39,6 +40,13 @@ _POSSIBLE_TARGET_ACTIONS = frozenset([
 _ENFORCEMENT_CHECK_ATTR = 'keystone:RBAC:enforcement_called'
 
 
+# TODO(gmann): Remove setting the default value of config policy_file
+# once oslo_policy change the default value to 'policy.yaml'.
+# https://github.com/openstack/oslo.policy/blob/a626ad12fe5a3abd49d70e3e5b95589d279ab578/oslo_policy/opts.py#L49
+DEFAULT_POLICY_FILE = 'policy.yaml'
+opts.set_defaults(CONF, DEFAULT_POLICY_FILE)
+
+
 class RBACEnforcer(object):
     """Enforce RBAC on API calls."""
 
diff --git a/keystone/conf/__init__.py b/keystone/conf/__init__.py
index 77c26a11d..5de0ec183 100644
--- a/keystone/conf/__init__.py
+++ b/keystone/conf/__init__.py
@@ -18,6 +18,7 @@ from oslo_log import log
 from oslo_log import versionutils
 import oslo_messaging
 from oslo_middleware import cors
+from oslo_policy import opts as policy_opts
 from osprofiler import opts as profiler
 
 from keystone.conf import application_credential
@@ -185,6 +186,12 @@ def set_external_opts_defaults():
     # configure OSprofiler options
     profiler.set_defaults(CONF, enabled=False, trace_sqlalchemy=False)
 
+    # TODO(gmann): Remove setting the default value of config policy_file
+    # once oslo_policy change the default value to 'policy.yaml'.
+    # https://github.com/openstack/oslo.policy/blob/a626ad12fe5a3abd49d70e3e5b95589d279ab578/oslo_policy/opts.py#L49
+    DEFAULT_POLICY_FILE = 'policy.yaml'
+    policy_opts.set_defaults(cfg.CONF, DEFAULT_POLICY_FILE)
+
     # Oslo.cache is always enabled by default for request-local caching
     # TODO(morganfainberg): Fix this to not use internal interface when
     # oslo.cache has proper interface to set defaults added. This is
diff --git a/keystone/tests/unit/test_policy.py b/keystone/tests/unit/test_policy.py
index 8346549ec..d58ea6399 100644
--- a/keystone/tests/unit/test_policy.py
+++ b/keystone/tests/unit/test_policy.py
@@ -197,7 +197,7 @@ class PolicyJsonTestCase(unit.TestCase):
     def test_all_targets_documented(self):
         policy_keys = self._get_default_policy_rules()
 
-        # These keys are in the policy.json but aren't targets.
+        # These keys are in the policy.yaml but aren't targets.
         policy_rule_keys = [
             'admin_or_owner', 'admin_or_token_subject', 'admin_required',
             'owner', 'service_admin_or_token_subject', 'service_or_admin',
diff --git a/keystone/tests/unit/test_v3.py b/keystone/tests/unit/test_v3.py
index 56ebf4a90..7d6c6b11f 100644
--- a/keystone/tests/unit/test_v3.py
+++ b/keystone/tests/unit/test_v3.py
@@ -247,7 +247,7 @@ class RestfulTestCase(unit.SQLDriverOverrides, rest.RestfulTestCase,
             domain_id=DEFAULT_DOMAIN_ID)
         self.default_domain_user_id = self.default_domain_user['id']
 
-        # create & grant policy.json's default role for admin_required
+        # create & grant policy.yaml's default role for admin_required
         self.role = unit.new_role_ref(name='admin')
         self.role_id = self.role['id']
         PROVIDERS.role_api.create_role(self.role_id, self.role)
diff --git a/lower-constraints.txt b/lower-constraints.txt
index 9e9220199..431589d02 100644
--- a/lower-constraints.txt
+++ b/lower-constraints.txt
@@ -25,16 +25,16 @@ oauthlib==0.6.2
 os-api-ref==1.4.0
 oslo.cache==1.26.0
 oslo.concurrency==3.26.0
-oslo.config==5.2.0
+oslo.config==6.8.0
 oslo.context==2.22.0
 oslo.db==6.0.0
 oslo.i18n==3.15.3
 oslo.log==3.44.0
 oslo.messaging==5.29.0
 oslo.middleware==3.31.0
-oslo.policy==3.0.2
+oslo.policy==3.6.0
 oslo.serialization==2.18.0
-oslo.upgradecheck==0.1.0
+oslo.upgradecheck==1.3.0
 oslo.utils==3.33.0
 oslotest==3.2.0
 osprofiler==1.4.0
diff --git a/releasenotes/notes/deprecate-json-formatted-policy-file-95f6307f88358f58.yaml b/releasenotes/notes/deprecate-json-formatted-policy-file-95f6307f88358f58.yaml
new file mode 100644
index 000000000..c9c530004
--- /dev/null
+++ b/releasenotes/notes/deprecate-json-formatted-policy-file-95f6307f88358f58.yaml
@@ -0,0 +1,20 @@
+---
+upgrade:
+  - |
+    The default value of ``[oslo_policy] policy_file`` config option has
+    been changed from ``policy.json`` to ``policy.yaml``.
+    Operators who are utilizing customized or previously generated
+    static policy JSON files (which are not needed by default), should
+    generate new policy files or convert them in YAML format. Use the
+    `oslopolicy-convert-json-to-yaml
+    <https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html>`_
+    tool to convert a JSON to YAML formatted policy file in
+    backward compatible way.
+deprecations:
+  - |
+    Use of JSON policy files was deprecated by the ``oslo.policy`` library
+    during the Victoria development cycle. As a result, this deprecation is
+    being noted in the Wallaby cycle with an anticipated future removal of support
+    by ``oslo.policy``. As such operators will need to convert to YAML policy
+    files. Please see the upgrade notes for details on migration of any
+    custom policy files.
diff --git a/requirements.txt b/requirements.txt
index 33a2c423e..3464fb3fc 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -16,16 +16,16 @@ keystonemiddleware>=7.0.0 # Apache-2.0
 bcrypt>=3.1.3 # Apache-2.0
 scrypt>=0.8.0 # BSD
 oslo.cache>=1.26.0 # Apache-2.0
-oslo.config>=5.2.0 # Apache-2.0
+oslo.config>=6.8.0 # Apache-2.0
 oslo.context>=2.22.0 # Apache-2.0
 oslo.messaging>=5.29.0 # Apache-2.0
 oslo.db>=6.0.0 # Apache-2.0
 oslo.i18n>=3.15.3 # Apache-2.0
 oslo.log>=3.44.0 # Apache-2.0
 oslo.middleware>=3.31.0 # Apache-2.0
-oslo.policy>=3.0.2 # Apache-2.0
+oslo.policy>=3.6.0 # Apache-2.0
 oslo.serialization!=2.19.1,>=2.18.0 # Apache-2.0
-oslo.upgradecheck>=0.1.0 # Apache-2.0
+oslo.upgradecheck>=1.3.0 # Apache-2.0
 oslo.utils>=3.33.0 # Apache-2.0
 oauthlib>=0.6.2 # BSD
 pysaml2>=5.0.0