New implementation of BLACKHOLEs

This replaces the global blackhole_queue with a clever scheme that
enables us to queue up blocked threads on the closure that they are
blocked on, while still avoiding atomic instructions in the common
case.

Advantages:

 - gets rid of a locked global data structure and some tricky GC code
   (replacing it with some per-thread data structures and different
   tricky GC code :)

 - wakeups are more prompt: parallel/concurrent performance should
   benefit.  I haven't seen anything dramatic in the parallel
   benchmarks so far, but a couple of threading benchmarks do improve
   a bit.

 - waking up a thread blocked on a blackhole is now O(1) (e.g. if
   it is the target of throwTo).

 - less sharing and better separation of Capabilities: communication
   is done with messages, the data structures are strictly owned by a
   Capability and cannot be modified except by sending messages.

 - this change will utlimately enable us to do more intelligent
   scheduling when threads block on each other.  This is what started
   off the whole thing, but it isn't done yet (#3838).

I'll be documenting all this on the wiki in due course.

1 files changed, 11 insertions, 8 deletions

diff --git a/rts/Capability.h b/rts/Capability.h
index 4030b5efd4..e12b8ce6e9 100644
--- a/rts/Capability.h
+++ b/rts/Capability.h
@@ -201,6 +201,8 @@ void waitForReturnCapability (Capability **cap/*in/out*/, Task *task);
 
 INLINE_HEADER void recordMutableCap (StgClosure *p, Capability *cap, nat gen);
 
+INLINE_HEADER void recordClosureMutated (Capability *cap, StgClosure *p);
+
 #if defined(THREADED_RTS)
 
 // Gives up the current capability IFF there is a higher-priority
@@ -222,12 +224,6 @@ void yieldCapability (Capability** pCap, Task *task);
 //
 void waitForCapability (Task *task, Mutex *mutex, Capability **pCap);
 
-// Wakes up a thread on a Capability (probably a different Capability
-// from the one held by the current Task).
-//
-void wakeupThreadOnCapability (Capability *my_cap, Capability *other_cap,
-                               StgTSO *tso);
-
 // Wakes up a worker thread on just one Capability, used when we
 // need to service some global event.
 //
@@ -289,8 +285,6 @@ void traverseSparkQueues (evac_fn evac, void *user);
 
 INLINE_HEADER rtsBool emptyInbox(Capability *cap);;
 
-void sendMessage (Capability *cap, Message *msg);
-
 #endif // THREADED_RTS
 
 /* -----------------------------------------------------------------------------
@@ -316,6 +310,15 @@ recordMutableCap (StgClosure *p, Capability *cap, nat gen)
     *bd->free++ = (StgWord)p;
 }
 
+INLINE_HEADER void
+recordClosureMutated (Capability *cap, StgClosure *p)
+{
+    bdescr *bd;
+    bd = Bdescr((StgPtr)p);
+    if (bd->gen_no != 0) recordMutableCap(p,cap,bd->gen_no);
+}
+
+
 #if defined(THREADED_RTS)
 INLINE_HEADER rtsBool
 emptySparkPoolCap (Capability *cap)