SERVER-8891 Destroy static objects in a safer order

Change the order of some static objects and group them near the
start of the source file, along with a comment explaining what is
going on.  The crashes we've seen have been due to destructors for
ReplicaSetMonitors (triggered by the destruction of _sets) trying
to use the _seedServers map, which had been destroyed already.  By
changing the order of the object definitions, we destroy _sets
before destroying _seedServers, preventing the crash.

There may be other cases that are not solved by this fix, and there
is still a race due to the running ReplicaSetMonitorWatcher thread,
so this is unlikely to be the last word on crashes of this type.

1 files changed, 8 insertions, 5 deletions

diff --git a/src/mongo/client/dbclient_rs.h b/src/mongo/client/dbclient_rs.h
index 2a1793e1ef0..9c6d191b12f 100644
--- a/src/mongo/client/dbclient_rs.h
+++ b/src/mongo/client/dbclient_rs.h
@@ -206,7 +206,7 @@ namespace mongo {
         /**
          * Removes the ReplicaSetMonitor for the given set name from _sets, which will delete it.
          * If clearSeedCache is true, then the cached seed string for this Replica Set will be removed
-         * from _setServers.
+         * from _seedServers.
          */
         static void remove( const string& name, bool clearSeedCache = false );
 
@@ -324,7 +324,7 @@ namespace mongo {
                 bool verbose, int nodesOffset );
 
         /**
-         * Save the seed list for the current set into the _setServers map
+         * Save the seed list for the current set into the _seedServers map
          * Should only be called if you're already holding _setsLock and this
          * monitor's _lock.
          */
@@ -388,9 +388,12 @@ namespace mongo {
         // The number of consecutive times the set has been checked and every member in the set was down.
         int _failedChecks;
 
-        static mongo::mutex _setsLock; // protects _sets and _setServers
-        static map<string,ReplicaSetMonitorPtr> _sets; // set name to Monitor
-        static map<string,vector<HostAndPort> > _seedServers; // set name to seed list. Used to rebuild the monitor if it is cleaned up but then the set is accessed again.
+        static mongo::mutex _setsLock; // protects _seedServers and _sets
+
+        // set name to seed list.
+        // Used to rebuild the monitor if it is cleaned up but then the set is accessed again.
+        static map<string, vector<HostAndPort> > _seedServers;
+        static map<string, ReplicaSetMonitorPtr> _sets; // set name to Monitor
 
         static ConfigChangeHook _hook;
         int _localThresholdMillis; // local ping latency threshold (protected by _lock)