QPID-2920: Updates to new-cluster-plan.

Filled out plan to-do list. More implementation detail.

git-svn-id: https://svn.apache.org/repos/asf/qpid/trunk@1186463 13f79535-47bb-0310-9956-ffa450edef68

2 files changed, 41 insertions, 18 deletions

diff --git a/qpid/cpp/design_docs/new-cluster-design.txt b/qpid/cpp/design_docs/new-cluster-design.txt
index a162ea68ec..936530a39a 100644
--- a/qpid/cpp/design_docs/new-cluster-design.txt
+++ b/qpid/cpp/design_docs/new-cluster-design.txt
@@ -83,20 +83,21 @@ context.
 
 ** A new cluster design.
 
-Clearly defined interface between broker code and cluster plug-in.
+1. Clearly defined interface between broker code and cluster plug-in.
 
-Replicate queue events rather than client data.
-- Broker behavior only needs to match per-queue.
-- Smaller amount of code (queue implementation) that must behave predictably.
-- Events only need be serialized per-queue, allows concurrency between queues
+2. Replicate queue events rather than client data.
+ - Only requires consistent enqueue order.
+ - Events only need be serialized per-queue, allows concurrency between queues
+ - Allows for replicated and non-replicated queues.
 
-Use a moving queue ownership protocol to agree order of dequeues.
-No longer relies on identical state and lock-step behavior to cause
-identical dequeues on each broker.
+3. Use a lock protocol to agree order of dequeues: only the broker
+   holding the lock can acqiure & dequeue.  No longer relies on
+   identical state and lock-step behavior to cause identical dequeues
+   on each broker.
 
-Use multiple CPG groups to process different queues in parallel. Use a
-fixed set of groups and hash queue names to choose the group for each
-queue.
+4. Use multiple CPG groups to process different queues in
+   parallel. Use a fixed set of groups and hash queue names to choose
+   the group for each queue.
 
 *** Requirements
 
diff --git a/qpid/cpp/design_docs/new-cluster-plan.txt b/qpid/cpp/design_docs/new-cluster-plan.txt
index 32e3f710e7..626e443be7 100644
--- a/qpid/cpp/design_docs/new-cluster-plan.txt
+++ b/qpid/cpp/design_docs/new-cluster-plan.txt
@@ -90,7 +90,9 @@ Independent message IDs that can be generated and sent with the message simplify
 this and potentially allow performance benefits by relaxing total ordering.
 However they imply additional map lookups that might hurt performance.
 
-- [ ] Prototype independent message IDs, check performance.
+- [X] Prototype independent message IDs, check performance.
+Throughput worse by 30% in contented case, 10% in uncontended.
+Sticking with queue sequence numbers.
 
 * Outstanding Tasks
 ** TODO [#A] Defer and async completion of wiring commands.
@@ -152,6 +154,10 @@ Status includes
 - persistent store state (clean, dirty)
 - cluster protocol version.
 
+** TODO [#B] Replace boost::hash with our own hash function.
+The hash function is effectively part of the interface so
+we need to be sure it doesn't change underneath us.
+
 ** TODO [#B] Persistent cluster support.
 Initial status protoocl to support persistent start-up (see existing code)
 
@@ -192,6 +198,16 @@ When this is fixed in the standalone broker, it should be fixed for cluster.
 ** TODO [#B] Network partitions and quorum.
 Re-use existing implementation.
 
+** TODO [#B] Review error handling, put in a consitent model.
+- [ ] Review all asserts, for possible throw.
+- [ ] Decide on fatal vs. non-fatal errors.
+
+** TODO [#B] Implement inconsistent error handling policy.
+What to do if a message is enqueued sucessfully on the local broker,
+but fails on one or more backups - e.g. due to store limits?
+- we have more flexibility, we don't *have* to crash
+- but we've loste some of our redundancy guarantee, how should we inform client? 
+
 ** TODO [#C] Allow non-replicated exchanges, queues.
 
 Set qpid.replicate=false in declare arguments, set flag on Exchange, Queue objects.
@@ -226,23 +242,29 @@ The old cluster has workarounds in the broker code that can be removed.
 - [ ] drop connections, sessions, management from cluster update.
 - [ ] drop security workarounds: cluster code now operates after message decoding.
 - [ ] drop connection tracking in cluster code.
-- [ ] simper inconsistent-error handling code, no need to stall.
+- [ ] simpler inconsistent-error handling code, no need to stall.
+
 ** TODO [#C] Support for live upgrades.
 
 Allow brokers in a running cluster to be replaced one-by-one with a new version.
+(see new-cluster-design for design notes.)
 
 The old cluster protocol was unstable because any changes in broker
 state caused changes to the cluster protocol.The new design should be
 much more stable.
 
-Points to implement: 
-- Brokers should ignore unknown controls (with a warning) rather than an error.
-- Limit logging frequency for unknown control warnings.
-- Add a version number at front of every CPG message. Determines how the
-  rest of the message is decoded. (allows for entirely new encodings e.g. AMQP 1.0)
+Points to implement in anticipation of live upgrade:
+- Prefix each CPG message with a version number and length.
+  Version number determines how to decode the message.
+- Brokers ignore messages that have a higher version number than they understand.
 - Protocol version XML element in cluster.xml, on each control.
 - Initial status protocol to include protocol version number.
 
+New member udpates: use the store encode/decode for updates, use the
+same backward compatibility strategy as the store. This allows for
+adding new elements to the end of structures but not changing or
+removing new elements.
+
 ** TODO [#C] Support for AMQP 1.0.
 
 * Testing