NonMoving: Don't do major GC if one is already running

Previously we would perform a preparatory moving collection, resulting
in many things being added to the mark queue. When we finished with this
we would realize in nonmovingCollect that there was already a collection
running, in which case we would simply not run the nonmoving collector.

However, it was very easy to end up in a "treadmilling" situation: all
subsequent GC following the first failed major GC would be scheduled as
major GCs. Consequently we would continuously feed the concurrent
collector with more mark queue entries and it would never finish.

This patch aborts the major collection far earlier, meaning that we
avoid adding nonmoving objects to the mark queue and allowing the
concurrent collector to finish.

1 files changed, 12 insertions, 0 deletions

diff --git a/rts/sm/GC.c b/rts/sm/GC.c
index a97042c718..6be81e5ff0 100644
--- a/rts/sm/GC.c
+++ b/rts/sm/GC.c
@@ -268,6 +268,18 @@ GarbageCollect (uint32_t collect_gen,
   /* See Note [Deadlock detection under nonmoving collector]. */
   deadlock_detect_gc = deadlock_detect;
 
+#if defined(THREADED_RTS)
+  if (major_gc && RtsFlags.GcFlags.useNonmoving && concurrent_coll_running) {
+      /* If there is already a concurrent major collection running then
+       * there is no benefit to starting another.
+       * TODO: Catch heap-size runaway.
+       */
+      N--;
+      collect_gen--;
+      major_gc = false;
+  }
+#endif
+
   /* N.B. The nonmoving collector works a bit differently. See
    * Note [Static objects under the nonmoving collector].
    */