dup2, dup3: work around another cygwin crasher

Cygwin 1.7.25 can crash due to an off-by-one bug on an attempt to
duplicate a file into the current RLIMIT_NOFILE soft limit, when
that limit is smaller than the hard limit.  The intent in the
cygwin code was to allow the dup and auto-increase the soft limit,
which is itself questionable (and which we work around in the
gnulib getdtablesize module); but avoiding the crash is worth
doing even if the soft limit semantics are wrong.

http://cygwin.com/ml/cygwin/2013-09/msg00397.html
http://cygwin.com/ml/cygwin-developers/2013-q3/msg00010.html

* m4/dup2.m4 (gl_FUNC_DUP2): Expose the bug.
* m4/dup3.m4 (gl_FUNC_DUP3): Likewise.
* tests/test-dup2.c (main): Likewise.
* lib/dup2.c (rpl_dup2): Use setdtablesize to avoid it.
* lib/dup3.c (dup3): Likewise.
* doc/posix-functions/dup2.texi (dup2): Document it.
* doc/glibc-functions/dup3.texi (dup3): Likewise.

Signed-off-by: Eric Blake <eblake@redhat.com>

2 files changed, 18 insertions, 0 deletions

diff --git a/tests/test-dup2.c b/tests/test-dup2.c
index ecb2692d2a..44039f102b 100644
--- a/tests/test-dup2.c
+++ b/tests/test-dup2.c
@@ -150,6 +150,15 @@ main (void)
   errno = 0;
   ASSERT (dup2 (fd, -2) == -1);
   ASSERT (errno == EBADF);
+  if (bad_fd > 256)
+    {
+      ASSERT (dup2 (fd, 255) == 255);
+      ASSERT (dup2 (fd, 256) == 256);
+      ASSERT (close (255) == 0);
+      ASSERT (close (256) == 0);
+    }
+  ASSERT (dup2 (fd, bad_fd - 1) == bad_fd - 1);
+  ASSERT (close (bad_fd - 1) == 0);
   errno = 0;
   ASSERT (dup2 (fd, bad_fd) == -1);
   ASSERT (errno == EBADF);
diff --git a/tests/test-dup3.c b/tests/test-dup3.c
index eb26c8f111..5a4aa6dd9e 100644
--- a/tests/test-dup3.c
+++ b/tests/test-dup3.c
@@ -124,6 +124,15 @@ main ()
       errno = 0;
       ASSERT (dup3 (fd, -2, o_flags) == -1);
       ASSERT (errno == EBADF);
+      if (bad_fd > 256)
+        {
+          ASSERT (dup3 (fd, 255, 0) == 255);
+          ASSERT (dup3 (fd, 256, 0) == 256);
+          ASSERT (close (255) == 0);
+          ASSERT (close (256) == 0);
+        }
+      ASSERT (dup3 (fd, bad_fd - 1, 0) == bad_fd - 1);
+      ASSERT (close (bad_fd - 1) == 0);
       errno = 0;
       ASSERT (dup3 (fd, bad_fd, o_flags) == -1);
       ASSERT (errno == EBADF);