Support for reducing the number of Capabilities with setNumCapabilities

This patch allows setNumCapabilities to /reduce/ the number of active
capabilities as well as increase it.  This is particularly tricky to
do, because a Capability is a large data structure and ties into the
rest of the system in many ways.  Trying to clean it all up would be
extremely error prone.

So instead, the solution is to mark the extra capabilities as
"disabled".  This has the following consequences:

  - threads on a disabled capability are migrated away by the
    scheduler loop

  - disabled capabilities do not participate in GC
    (see scheduleDoGC())

  - No spark threads are created on this capability
    (see scheduleActivateSpark())

  - We do not attempt to migrate threads *to* a disabled
    capability (see schedulePushWork()).

So a disabled capability should do no work, and does not participate
in GC, although it remains alive in other respects.  For example, a
blocked thread might wake up on a disabled capability, and it will get
quickly migrated to a live capability.  A disabled capability can
still initiate GC if necessary.  Indeed, it turns out to be hard to
migrate bound threads, so we wait until the next GC to do this (see
comments for details).

1 files changed, 17 insertions, 2 deletions

diff --git a/rts/Capability.c b/rts/Capability.c
index 41efb176fd..d04d007006 100644
--- a/rts/Capability.c
+++ b/rts/Capability.c
@@ -34,6 +34,7 @@
 Capability MainCapability;
 
 nat n_capabilities = 0;
+nat enabled_capabilities = 0;
 Capability *capabilities = NULL;
 
 // Holds the Capability which last became free.  This is used so that
@@ -323,6 +324,8 @@ initCapabilities( void )
 
 #endif
 
+    enabled_capabilities = n_capabilities;
+
     // There are no free capabilities to begin with.  We will start
     // a worker Task to each Capability, which will quickly put the
     // Capability on the free list when it finds nothing to do.
@@ -493,7 +496,7 @@ releaseCapability_ (Capability* cap,
     // anything else to do, give the Capability to a worker thread.
     if (always_wakeup || 
         !emptyRunQueue(cap) || !emptyInbox(cap) ||
-        !emptySparkPoolCap(cap) || globalWorkToDo()) {
+        (!cap->disabled && !emptySparkPoolCap(cap)) || globalWorkToDo()) {
 	if (cap->spare_workers) {
 	    giveCapabilityToTask(cap,cap->spare_workers);
 	    // The worker Task pops itself from the queue;
@@ -682,7 +685,8 @@ yieldCapability (Capability** pCap, Task *task)
         gcWorkerThread(cap);
         traceEventGcEnd(cap);
         traceSparkCounters(cap);
-        return;
+        // See Note [migrated bound threads 2]
+        if (task->cap == cap) return;
     }
 
 	debugTrace(DEBUG_sched, "giving up capability %d", cap->no);
@@ -768,6 +772,17 @@ yieldCapability (Capability** pCap, Task *task)
 //    hold Capabilty C, and task->cap == C, then task cannot be
 //    migrated under our feet.
 
+// Note [migrated bound threads 2]
+//
+// Second tricky case;
+//   - A bound Task becomes a GC thread
+//   - scheduleDoGC() migrates the thread belonging to this Task,
+//     because the Capability it is on is disabled
+//   - after GC, gcWorkerThread() returns, but now we are
+//     holding a Capability that is not the same as task->cap
+//   - Hence we must check for this case and immediately give up the
+//     cap we hold.
+
 /* ----------------------------------------------------------------------------
  * prodCapability
  *