rts: Ensure we always give MADV_DONTNEED a chance in osDecommitMemory

As described in #12865, newer Linux kernels support both MADV_FREE and
MADV_DONTNEED. Previously a runtime would fail to try MADV_DONTNEED if
MADV_FREE failed (e.g. since the kernel which the image is running on
doesn't support the latter). Now we try MADV_DONTNEED if MADV_FREE
failed to ensure that binaries compiled on a kernel supporting MADV_FREE
don't fail on decommit.

Test Plan: Validate

Reviewers: austin, erikd, simonmar

Reviewed By: simonmar

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D2780

GHC Trac Issues: #12865

1 files changed, 16 insertions, 3 deletions

diff --git a/rts/posix/OSMem.c b/rts/posix/OSMem.c
index 5291745d53..beffedaac4 100644
--- a/rts/posix/OSMem.c
+++ b/rts/posix/OSMem.c
@@ -541,11 +541,24 @@ void osDecommitMemory(void *at, W_ size)
 
 #ifdef MADV_FREE
     // Try MADV_FREE first, FreeBSD has both and MADV_DONTNEED
-    // just swaps memory out
+    // just swaps memory out. Linux >= 4.5 has both DONTNEED and FREE; either
+    // will work as they both allow the system to free anonymous pages.
+    // It is important that we try both methods as the kernel which we were
+    // built on may differ from the kernel we are now running on.
     r = madvise(at, size, MADV_FREE);
-#else
-    r = madvise(at, size, MADV_DONTNEED);
+    if(r < 0) {
+        if (errno == EINVAL) {
+            // Perhaps the system doesn't support MADV_FREE; fall-through and
+            // try MADV_DONTNEED.
+        } else {
+            sysErrorBelch("unable to decommit memory");
+        }
+    } else {
+        return;
+    }
 #endif
+
+    r = madvise(at, size, MADV_DONTNEED);
     if(r < 0)
         sysErrorBelch("unable to decommit memory");
 }