Increase the default max_overflow value

It turned out that distribution of WSGI requests between forks is not
even when eventlet is used, which can cause pool timeout issues under
load, while there are idle workers with open DB connections, which
should have served these HTTP requests.

The default max_overflow value should be increased to allow DB oriented
services (mostly APIs) handle spikes in number of concurrent requests
trying to use a DB.

At the same time, the default number of greenlets is decreased:

 I65b40b9906b75146a0085bbe168f1e6bcae82f21

which effectively causes a particular worker to stop accepting new
requests it has no resources (DB connections) to handle and allow
other forks accept()'ing on the same FD to proceed.

Testing was performed using this script:

  https://gist.github.com/zzzeek/c69138fd0d0b3e553a1f

With 100 greenthreads in the pool, 50 DB connections allowed overflow
no pool timeout issues were seen with up to 500 concurrent requests
done by ab, while current default values (1000/10) could not handle
even 100 concurrent requests.

See this ML thread for details:

  http://lists.openstack.org/pipermail/openstack-dev/2015-December/082717.html

DocImpact

This change potentially increases the number of connections open to the
RDBMS server. If you start seeing "too many connections" errors, please
check these settings and adjust them appropriately:

https://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html#sysvar_max_connections
http://www.postgresql.org/docs/current/static/runtime-config-connection.html#GUC-MAX-CONNECTIONS

Closes-Bug: #1535375

Change-Id: I2e9c2a71d8231e0dfbefc6293ad319e1e459beec

2 files changed, 26 insertions, 0 deletions

diff --git a/oslo_db/options.py b/oslo_db/options.py
index 38e9024..f4bec25 100644
--- a/oslo_db/options.py
+++ b/oslo_db/options.py
@@ -89,6 +89,7 @@ database_opts = [
                                                   group='DATABASE')],
                help='Interval between retries of opening a SQL connection.'),
     cfg.IntOpt('max_overflow',
+               default=50,
                deprecated_opts=[cfg.DeprecatedOpt('sql_max_overflow',
                                                   group='DEFAULT'),
                                 cfg.DeprecatedOpt('sqlalchemy_max_overflow',
diff --git a/releasenotes/notes/increase-default-max-overflow-0af787268807f926.yaml b/releasenotes/notes/increase-default-max-overflow-0af787268807f926.yaml
new file mode 100644
index 0000000..6547ac2
--- /dev/null
+++ b/releasenotes/notes/increase-default-max-overflow-0af787268807f926.yaml
@@ -0,0 +1,25 @@
+---
+upgrade:
+  - |
+    The default value of ``max_overflow`` config option
+    has been increased from 10 to 50 in order to allow
+    OpenStack services heavily using DBs to better handle
+    spikes of concurrent requests and lower the probability
+    of getting a pool timeout issue.
+
+    This change potentially leads to increasing of the number
+    of open connections to an RDBMS server. Depending on the
+    configuration, you may see "too many connections" errors
+    in logs of OpenStack services / RDBMS server. The max limit of
+    connections can be set by the means of these config options:
+
+    http://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_max_connections
+    http://www.postgresql.org/docs/current/static/runtime-config-connection.html#GUC-MAX-CONNECTIONS
+
+    For details, please see the following LP:
+
+    https://bugs.launchpad.net/oslo.db/+bug/1535375
+
+    and the ML thread:
+
+    http://lists.openstack.org/pipermail/openstack-dev/2015-December/082717.html