objectcacher: don't wait for write waiters; wait after dirtying

We do three things here:

- Wait for the dirty limit to drop _after_ writing into the cache.  This
  means that an active thread can always provide its dirty data to the
  cache for potential writing without waiting (a small win).  It's also
  helpful later... (see below, and next commit)

- Don't wait for other waiters.  If another thread dirtying 1MB and is
  waiting for it, don't wait for them too.  This prevents two threads
  writing 1MB at a time with a limit of 1MB from serializing: both can
  dirty their 1MB and initiate a flush, and they once 1/2 of that has
  flushed one of them will be allowed to proceed.

- Update the flusher to add the dirty_waiting bytes to the amount to
  write so that the OPs will indeed be parallel.

Signed-off-by: Sage Weil <sage@newdream.net>

1 files changed, 1 insertions, 1 deletions

diff --git a/src/librbd.cc b/src/librbd.cc
index bb7043682aa..57a4eb8cfc9 100644
--- a/src/librbd.cc
+++ b/src/librbd.cc
@@ -288,8 +288,8 @@ namespace librbd {
       wr->extents.push_back(extent);
       {
 	Mutex::Locker l(cache_lock);
-	object_cacher->wait_for_write(len, cache_lock);
 	object_cacher->writex(wr, object_set);
+	object_cacher->wait_for_write(len, cache_lock);
       }
     }