build: Fix broken mlock detection

* acinclude.m4 [GNUPG_CHECK_MLOCK]: Use size_t for the ptr test.
--

We need to be careful when casting a pointer to a `long int`: the
highest bit might be set, in which case the result is a negative number.

In this instance, it is fatal: we now take the modulus of that negative
number with regards to the page size, and subtract it from the page
size. So what should be a number that is smaller than the page size is
now larger than the page size.

As a consequence, we do not try to lock a 4096-byte block that is at the
page size boundary inside a `malloc()`ed block, but we try to do that
_outside_ the block.

Which means that we are not at all detecting whether `mlock()` is
broken.

This actually happened here, in the i686 MSYS2 build of libgcrypt.

Let's be very careful to case the pointer to an _unsigned_ value
instead.

Note: technically, we should cast the pointer to a `size_t`. But since
we only need the remainder modulo the page size (which is a power of
two) anyway, it does not matter whether we clip, say, a 64-bit `size_t`
to a 32-bit `unsigned long`. It does matter, though, whether we
mistakenly turn the remainder into a negative one.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

- Changed to use size_t for clarity.
- Added ChangeLog entry.

Take care: Building for Windows using MSYS is not supported, please
use gcc and cross-compile.

Signed-off-by: Werner Koch <wk@gnupg.org>

1 files changed, 1 insertions, 1 deletions

diff --git a/acinclude.m4 b/acinclude.m4
index 3c8dfba7..05bf88a7 100644
--- a/acinclude.m4
+++ b/acinclude.m4
@@ -236,7 +236,7 @@ int main()
     pool = malloc( 4096 + pgsize );
     if( !pool )
         return 2;
-    pool += (pgsize - ((long int)pool % pgsize));
+    pool += (pgsize - ((size_t)pool % pgsize));
 
     err = mlock( pool, 4096 );
     if( !err || errno == EPERM || errno == EAGAIN)