rts/posix: Reduce heap allocation amount on mmap failure

Since the two-step allocator the RTS asks the kernel for a large upfront
mmap'd region of memory (on the order of terabytes). While we have no
expectation that this entire region will be backed by physical memory,
this scheme nevertheless fails on some systems with resource limits.
Here we use a back-off scheme to reduce our allocation request until we
find a size agreeable to the kernel. Fixes #10877.

This also fixes a latent bug wherein the heap reservation retry logic
would fail to free the previously reserved address space, which would
likely result in a heap allocation failure.

Test Plan:
set address space limit with `ulimit -v 67108864` and try running
a compiled program

Reviewers: simonmar, austin

Reviewed By: simonmar

Subscribers: thomie, RyanGlScott

Differential Revision: https://phabricator.haskell.org/D1405

GHC Trac Issues: #10877

1 files changed, 5 insertions, 1 deletions

diff --git a/rts/sm/OSMem.h b/rts/sm/OSMem.h
index 6bcaf65b10..533f6f7fe6 100644
--- a/rts/sm/OSMem.h
+++ b/rts/sm/OSMem.h
@@ -34,8 +34,12 @@ void setExecutable (void *p, W_ len, rtsBool exec);
 // pointed to by the return value, until that memory is committed using
 // osCommitMemory().
 //
+// The value pointed to by len will be filled by the caller with an upper
+// bound on the amount of memory to reserve. On return this will be set
+// to the amount of memory actually reserved.
+//
 // This function is called once when the block allocator is initialized.
-void *osReserveHeapMemory(W_ len);
+void *osReserveHeapMemory(W_ *len);
 
 // Commit (allocate memory for) a piece of address space, which must
 // be within the previously reserved space After this call, it is safe