msgr: drop messages on cons with CLOSED Pipes

Back in commit 6339c5d43974f4b495f15d199e01a141e74235f5, we tried to make
this deal with a race between a faulting pipe and new messages being
queued.  The sequence is

- fault starts on pipe
- fault drops pipe_lock to unregister the pipe
- user (objecter) queues new message on the con
- submit_message reopens a Pipe (due to this bug)
- the message managed to make it out over the wire
- fault finishes faulting, calls ms_reset
- user (objecter) closes the con
- user (objecter) resends everything

It appears as though the previous patch *meant* to drop *m on the floor in
this case, which is what this patch does.  And that fixes the crash I am
hitting; see #4271.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

1 files changed, 2 insertions, 0 deletions

diff --git a/src/msg/SimpleMessenger.cc b/src/msg/SimpleMessenger.cc
index dd496947151..e3b378046d2 100644
--- a/src/msg/SimpleMessenger.cc
+++ b/src/msg/SimpleMessenger.cc
@@ -411,6 +411,8 @@ void SimpleMessenger::submit_message(Message *m, Connection *con,
       pipe->put();
       ldout(cct,20) << "submit_message " << *m << " remote, " << dest_addr
 		    << ", had pipe " << pipe << ", but it closed." << dendl;
+      m->put();
+      return;
     }
   }