Qpid: advance thru the list of brokers on reconnect

Sync 877f1cab328fe8a9f8214687ea7e1287862de87d from oslo-incubator's
stable/havana branch.

In Qpid implementation, when using multiple qpid_hosts, we don't want to
immediately retry failed connection for the same failed broker. This was not
the case in existing implementation though, where we've always attempted to
reconnect starting from the first broker in the list of candidates. So if the
first broker failed, we initiated reconnect to the same failed broker.

This change makes reconnect() implementation to select the next broker in the
list. This also means that non-failure reconnect attempts will also switch the
current broker. All in all, users should not rely on any particular order to
use brokers from the list, so this should not constitute an issue.

Change-Id: I5a630e76880c0d4219c4a8a2b3354e41899d9695
Partial-Bug: 1261631

1 files changed, 5 insertions, 3 deletions

diff --git a/heat/openstack/common/rpc/impl_qpid.py b/heat/openstack/common/rpc/impl_qpid.py
index cc44b48f2..31c96bc34 100644
--- a/heat/openstack/common/rpc/impl_qpid.py
+++ b/heat/openstack/common/rpc/impl_qpid.py
@@ -468,6 +468,10 @@ class Connection(object):
         self.brokers = params['qpid_hosts']
         self.username = params['username']
         self.password = params['password']
+
+        brokers_count = len(self.brokers)
+        self.next_broker_indices = itertools.cycle(range(brokers_count))
+
         self.connection_create(self.brokers[0])
         self.reconnect()
 
@@ -495,7 +499,6 @@ class Connection(object):
 
     def reconnect(self):
         """Handles reconnecting and re-establishing sessions and queues."""
-        attempt = 0
         delay = 1
         while True:
             # Close the session if necessary
@@ -505,8 +508,7 @@ class Connection(object):
                 except qpid_exceptions.ConnectionError:
                     pass
 
-            broker = self.brokers[attempt % len(self.brokers)]
-            attempt += 1
+            broker = self.brokers[next(self.next_broker_indices)]
 
             try:
                 self.connection_create(broker)