Fix deadlock (#10545)

yieldCapability() was not prepared to be called by a Task that is not
either a worker or a bound Task.  This could happen if we ended up in
yieldCapability via this call stack:

performGC()
scheduleDoGC()
requestSync()
yieldCapability()

and there were a few other ways this could happen via requestSync.
The fix is to handle this case in yieldCapability(): when the Task is
not a worker or a bound Task, we put it on the returning_workers
queue, where it will be woken up again.

Summary of changes:

* `yieldCapability`: factored out subroutine waitForWorkerCapability`
* `waitForReturnCapability` renamed to `waitForCapability`, and
  factored out subroutine `waitForReturnCapability`
* `releaseCapabilityAndQueue` worker renamed to `enqueueWorker`, does
  not take a lock and no longer tests if `!isBoundTask()`
* `yieldCapability` adjusted for refactorings, only change in behavior
  is when it is not a worker or bound task.

Test Plan:
* new test concurrent/should_run/performGC
* validate

Reviewers: niteria, austin, ezyang, bgamari

Subscribers: thomie, bgamari

Differential Revision: https://phabricator.haskell.org/D997

GHC Trac Issues: #10545

1 files changed, 11 insertions, 0 deletions

diff --git a/rts/Task.h b/rts/Task.h
index 5c7b049631..58798bd24d 100644
--- a/rts/Task.h
+++ b/rts/Task.h
@@ -167,6 +167,17 @@ isBoundTask (Task *task)
     return (task->incall->tso != NULL);
 }
 
+// A Task is currently a worker if
+//  (a) it was created as a worker (task->worker), and
+//  (b) it has not left and re-entered Haskell, in which case
+//      task->incall->prev_stack would be non-NULL.
+//
+INLINE_HEADER rtsBool
+isWorker (Task *task)
+{
+    return (task->worker && task->incall->prev_stack == NULL);
+}
+
 // Linked list of all tasks.
 //
 extern Task *all_tasks;