Make clearNursery free

Summary:
clearNursery resets all the bd->free pointers of nursery blocks to
make the blocks empty.  In profiles we've seen clearNursery taking
significant amounts of time particularly with large -N and -A values.

This patch moves the work of clearNursery to the point at which we
actually need the new block, thereby introducing an invariant that
blocks to the right of the CurrentNursery pointer still need their
bd->free pointer reset.  This should make things faster overall,
because we don't need to clear blocks that we don't use.

Test Plan: validate

Reviewers: AndreasVoellmy, ezyang, austin

Subscribers: thomie, carter, ezyang, simonmar

Differential Revision: https://phabricator.haskell.org/D318

1 files changed, 7 insertions, 11 deletions

diff --git a/rts/Schedule.c b/rts/Schedule.c
index 6a06f792ea..447b70ef52 100644
--- a/rts/Schedule.c
+++ b/rts/Schedule.c
@@ -1125,21 +1125,16 @@ scheduleHandleHeapOverflow( Capability *cap, StgTSO *t )
 
         // don't do this if the nursery is (nearly) full, we'll GC first.
         if (cap->r.rCurrentNursery->link != NULL ||
-            cap->r.rNursery->n_blocks == 1) {  // paranoia to prevent infinite loop
-                                               // if the nursery has only one block.
+            cap->r.rNursery->n_blocks == 1) {  // paranoia to prevent
+                                               // infinite loop if the
+                                               // nursery has only one
+                                               // block.
 
             bd = allocGroup_lock(blocks);
             cap->r.rNursery->n_blocks += blocks;
 
-            // link the new group into the list
-            bd->link = cap->r.rCurrentNursery;
-            bd->u.back = cap->r.rCurrentNursery->u.back;
-            if (cap->r.rCurrentNursery->u.back != NULL) {
-                cap->r.rCurrentNursery->u.back->link = bd;
-            } else {
-                cap->r.rNursery->blocks = bd;
-            }
-            cap->r.rCurrentNursery->u.back = bd;
+            // link the new group after CurrentNursery
+            dbl_link_insert_after(bd, cap->r.rCurrentNursery);
 
             // initialise it as a nursery block.  We initialise the
             // step, gen_no, and flags field of *every* sub-block in
@@ -1162,6 +1157,7 @@ scheduleHandleHeapOverflow( Capability *cap, StgTSO *t )
             IF_DEBUG(sanity, checkNurserySanity(cap->r.rNursery));
 
             // now update the nursery to point to the new block
+            finishedNurseryBlock(cap, cap->r.rCurrentNursery);
             cap->r.rCurrentNursery = bd;
 
             // we might be unlucky and have another thread get on the