Switch to eventlet-safe oslo.cache's MemcacheClientPool

In past days there were discussions about various issues
with memcached connections [1][2][3].

After investigation it looks like common root cause for above
problems is keystonemiddleware. More precisely said the way
how keystonemiddleware is caching tokens.

Currently it's using some home-made CachePool with direct
usage of memcached library, moreover it looks like its
approach is not eventlet-safe.
Discussion can be mainly found in [4].

Fortunately keystonemiddleware can use "advanced cache pool",
which is oslo.cache's implementation and was added long time ago [5],
but it is turned on only if memcache_use_advanced_pool=True.

This patch is switching to more elaborated oslo.cache CachePool
and adding deprecation warning about eventlet-unsafe variant
of keystonemiddleware's memcache pool.

How to reproduce ?

with memcache_use_advanced_pool=False

1. Build clean ENV of openstack
2. Deploy core projects (keystone,glance,nova,placement...)
3. Run while true; do COMMAND FOR SERVICE; done
   - several bashes, in parallel (5-7)

COMMAND FOR SERVICE:
- openstack network list
- openstack volume list
- openstack server list
- openstack image list

4. Check memcached connections (which will grow up):
    - ss | grep 11211 | wc -l   every second

How to fix and test it ?

Repeat above, to fix:
 - with memcache_use_advanced_pool=True
   OR
 - apply this patch

Compare measurements in graph.

[1] https://bugs.launchpad.net/keystonemiddleware/+bug/1892852
[2] https://bugs.launchpad.net/oslo.cache/+bug/1888394
[3] https://bugs.launchpad.net/keystonemiddleware/+bug/1883659

[4] https://review.opendev.org/c/openstack/oslo.cache/+/742193

[5] https://review.opendev.org/c/openstack/keystonemiddleware/+/268664

Closes-Bug: #1883659
Closes-Bug: #1892852
Closes-Bug: #1888394

Change-Id: I0e96334b65a0bf369ebf1d88651d13feb8d2ecac

1 files changed, 22 insertions, 0 deletions

diff --git a/releasenotes/notes/deprecate-eventlet-unsafe-memcacheclientpool-f8b4a6733513d73e.yaml b/releasenotes/notes/deprecate-eventlet-unsafe-memcacheclientpool-f8b4a6733513d73e.yaml
new file mode 100644
index 0000000..2269994
--- /dev/null
+++ b/releasenotes/notes/deprecate-eventlet-unsafe-memcacheclientpool-f8b4a6733513d73e.yaml
@@ -0,0 +1,22 @@
+---
+deprecations:
+  - |
+    We no longer recommend using the eventlet unsafe keystonemiddleware's
+    memcacheclientpool. This implementation may result in growing connections
+    to memcached.
+
+    It is recommended that the ``memcache_use_advanced_pool`` option
+    is set to ``True`` in the ``keystone_authtoken`` configuration section of
+    the various services (e.g. nova, glance, ...) when memcached is used for
+    token cache.
+upgrade:
+  - |
+    [`bug 1892852 <https://bugs.launchpad.net/keystonemiddleware/+bug/1892852>`_]
+    [`bug 1888394 <https://bugs.launchpad.net/oslo.cache/+bug/1888394>`_]
+    [`bug 1883659 <https://bugs.launchpad.net/keystonemiddleware/+bug/1883659>`_]
+    Keystonemiddleware now using eventlet-safe implementation of
+    ``MemcacheClientPool`` from oslo.cache's library by default.
+    The ``keystonemiddleware`` implementation is now deprecated. For backwards
+    compatibility, the ``[keystone_authtoken] memcache_use_advanced_pool``
+    option can be set to ``False`` config files of the various services (e.g.
+    nova, glance, ...) when memcached is used for token cache.