QPID-3121: Cluster management inconsistency when using persistent store.

With the store doing async completions, completion IO callbacks could
be queued differently on different nodes. This led to inconsistent
management changes in a cluster when a connection was modified in an
IO callback.

Fix was to mark IO callback processing as not cluster safe, so
connections don't record management stats during an IO callback.

Test changes:
- enable durable tests in test_management.
- add substitutions to mask known issue of inconsistent "stats changed" messages.
- add transactional client to test_management.
- ignore heartbeat connection close logs in cluster_test_logs.py
- make brokertest.retry more accurate
- fix minor bug in brokertest.log_ready.

git-svn-id: https://svn.apache.org/repos/asf/qpid/trunk@1078947 13f79535-47bb-0310-9956-ffa450edef68

1 files changed, 10 insertions, 3 deletions

diff --git a/qpid/cpp/src/tests/cluster_test_logs.py b/qpid/cpp/src/tests/cluster_test_logs.py
index 25ddb3b74c..9f7d1e2f6c 100755
--- a/qpid/cpp/src/tests/cluster_test_logs.py
+++ b/qpid/cpp/src/tests/cluster_test_logs.py
@@ -61,10 +61,10 @@ def filter_log(log):
         'warning CLOSING .* unsent data',
         'Inter-broker link ',
         'Running in a cluster, marking store',
-        'debug Sending keepalive signal to watchdog',
-        'last broker standing joined by 1 replicas, updating queue policies.'
+        'debug Sending keepalive signal to watchdog', # Watchdog timer thread
+        'last broker standing joined by 1 replicas, updating queue policies.',
+        'Connection .* timed out: closing' # heartbeat connection close
         ])
-    skip_re = re.compile(skip)
     # Regex to match a UUID
     uuid='\w\w\w\w\w\w\w\w-\w\w\w\w-\w\w\w\w-\w\w\w\w-\w\w\w\w\w\w\w\w\w\w\w\w'
     # Substitutions to remove expected differences
@@ -82,6 +82,13 @@ def filter_log(log):
         (r' map={.*_object_name:([^,}]*)[,}].*', r' \1'), # V2 map - just keep name
         (r'\d+-\d+-\d+--\d+', 'X-X-X--X'), # V1 Object IDs
         ]
+    # Substitutions to mask known issue: durable test shows inconsistent "changed stats for com.redhat.rhm.store:journal" messages.
+    skip += '|Changed V[12] statistics com.redhat.rhm.store:journal'
+    subs += [(r'to=console.obj.1.0.com.redhat.rhm.store.journal props=\d+ stats=\d+',
+              'to=console.obj.1.0.com.redhat.rhm.store.journal props=NN stats=NN')]
+
+    skip_re = re.compile(skip)
+    subs = [(re.compile(pattern), subst) for pattern, subst in subs]
     for l in open(log):
         if skip_re.search(l): continue
         for pattern,subst in subs: l = re.sub(pattern,subst,l)