threadStackOverflow: check whether stack squeezing released some stack (#3677)

In a stack overflow situation, stack squeezing may reduce the stack
size, but we don't know whether it has been reduced enough for the
stack check to succeed if we try again.  Fortunately stack squeezing
is idempotent, so all we need to do is record whether *any* squeezing
happened.  If we are at the stack's absolute -K limit, and stack
squeezing happened, then we try running the thread again.

We also want to avoid enlarging the stack if squeezing has already
released some of it.  However, we don't want to get into a
pathalogical situation where a thread has a nearly full stack (near
its current limit, but not near the absolute -K limit), keeps
allocating a little bit, squeezing removes a little bit, and then it
runs again.  So to avoid this, if we squeezed *and* there is still
less than BLOCK_SIZE_W words free, then we enlarge the stack anyway.

1 files changed, 5 insertions, 0 deletions

diff --git a/rts/ThreadPaused.c b/rts/ThreadPaused.c
index 58c30e330e..96a23673a7 100644
--- a/rts/ThreadPaused.c
+++ b/rts/ThreadPaused.c
@@ -317,5 +317,10 @@ end:
     if (RtsFlags.GcFlags.squeezeUpdFrames == rtsTrue &&
 	((weight <= 5 && words_to_squeeze > 0) || weight < words_to_squeeze)) {
 	stackSqueeze(tso, (StgPtr)frame);
+        tso->flags |= TSO_SQUEEZED;
+        // This flag tells threadStackOverflow() that the stack was
+        // squeezed, because it may not need to be expanded.
+    } else {
+        tso->flags &= ~TSO_SQUEEZED;
     }
 }