Upgrade GCC to 5.2.0 for Windows x86 and x86_64

This patch does a few things

- Moved GHC x86 to MinGW-w64 (Using Awson's patch)
- Moves Both GHCs to MSYS2 toolchains
- Completely removes the dependencies on the git tarball repo
  - Downloads only the required tarball for the architecture for
    which we are building
  - Downloads the perl tarball is missing as well
  - Fixed a few bugs in the linker to fix tests on Windows

The links currently point to repo.msys2.org and GitHub, it might be
more desirable to mirror them on
http://downloads.haskell.org/~ghc/mingw/ as with the previous patch
attempt.

For more details on what the MSYS2 packages I include see #10726
(Awson's comment). but it should contain all we need
and no python or fortran, which makes the uncompressed tar a 1-2
hundreds mb smaller.

The `GCC 5.2.0` in the package supports `libgcc` as a shared library,
this is a problem since
when compiling with -shared the produced dll now has a dependency on
`libgcc_s_sjlj-1.dll`.
To solve this the flag `-static-libgcc` is now being used for all GCC
calls on windows.

Test Plan:
./validate was ran both on x86 and x86_64 windows and compared against
the baseline.

A few test were failing due to Ld no longer being noisy. These were
updated.

The changes to the configure script *should* be validated by the build
bots for the other platforms before landing

Reviewers: simonmar, awson, bgamari, austin, thomie

Reviewed By: thomie

Subscribers: #ghc_windows_task_force, thomie, awson

Differential Revision: https://phabricator.haskell.org/D1123

GHC Trac Issues: #10726, #9014, #9218, #10435

1 files changed, 112 insertions, 70 deletions

diff --git a/configure.ac b/configure.ac
index 9b21d07442..a46110be18 100644
--- a/configure.ac
+++ b/configure.ac
@@ -286,96 +286,135 @@ fail() {
     exit 1
 }
 
-set_up_tarballs() {
-    local tarball_repo_url="$1"
-    local tarball_dir="$2"
-
-    if ! test -d "${tarball_dir}"
+download_file() {
+    local file_url="$1"
+    local file_md5="$2"
+    local dest_file="$3"
+    local description="$4"
+    local extra_curl_opts="$5"
+    local dest_dir="$(dirname $dest_file)"
+
+    if ! test -f "${dest_file}"
     then
-        local git_cmd="git clone ${tarball_repo_url} ${tarball_dir}"
+        local curl_cmd="curl -L ${file_url} -o ${dest_file} --create-dirs -# ${extra_curl_opts}"
         if test "$TarballsAutodownload" = "NO"
         then
             echo >&2
-            echo "ERROR: Windows tarball binary distributions not found." >&2
-            echo "Please rerun configure with --enable-tarballs-autodownload, or clone the repository manually:" >&2
-            echo "  $git_cmd" >&2
+            echo "ERROR: ${description} not found." >&2
+            echo "Please rerun configure with --enable-tarballs-autodownload, or perform the download manually:" >&2
+            echo "  $curl_cmd" >&2
             exit 1
         fi
-        AC_MSG_NOTICE([Downloading Windows tarball distributions to ${tarball_dir}...])
-        $git_cmd || {
-            rm -f "${tarball_dir}"
-            fail "ERROR: Git clone failed."
+        AC_MSG_NOTICE([Downloading ${description} to ${dest_dir}...])
+        $curl_cmd || {
+            rm -f "${dest_file}"
+            fail "ERROR: Download failed."
         }
     else
-        AC_MSG_NOTICE([Using Windows tarball distributions found in ${tarball_dir}...])
+        AC_MSG_NOTICE([Using ${description} found in ${dest_dir}...])
     fi
+
+    echo "${file_md5} *${dest_file}" | md5sum --quiet -c - ||
+        fail "ERROR: ${description} appears to be corrupted, please delete it and try again."
 }
 
+
 if test "$HostOS" = "mingw32"
 then
-    test -d inplace || mkdir inplace
+    # Find the mingw-w64 7z file to extract.
+    # NB. If you update the tarballs to a new version of gcc, don't
+    # forget to tweak the paths in driver/gcc/gcc.c.
+    if test "$HostArch" = "i386"
+    then
+        mingw_arch="i686"
+        tarball_dest_dir="mingw-w64/x86"
+        tarball_mingw_dir="mingw32"
+    else
+        mingw_arch="x86_64"
+        tarball_dest_dir="mingw-w64/x86_64"
+        tarball_mingw_dir="mingw64"
+    fi
 
-    # NB. For now just run git clone on the tarball repo
-    ghc_tarball_repo='git://git.haskell.org/ghc-tarballs.git'
-    ghc_tarball_dir='ghc-tarballs'
-    set_up_tarballs "${ghc_tarball_repo}" "${ghc_tarball_dir}"
+    tarball_dir='ghc-tarballs'
+fi
+
+download_and_extract() {
+    local mingw_url="$1"
+    local file_md5sum_x86="$2"
+    local file_md5sum_x64="$3"
 
     if test "$HostArch" = "i386"
     then
-
-        # NB. If you update the tarballs to a new version of gcc, don't
-        # forget to tweak the paths in driver/gcc/gcc.c.
-        if ! test -d inplace/mingw ||
-             test inplace/mingw -ot ghc-tarballs/mingw/binutils*.tar.lzma  ||
-             test inplace/mingw -ot ghc-tarballs/mingw/gcc-core*.tar.lzma  ||
-             test inplace/mingw -ot ghc-tarballs/mingw/gcc-c++*.tar.lzma   ||
-             test inplace/mingw -ot ghc-tarballs/mingw/libgcc*.tar.gz      ||
-             test inplace/mingw -ot ghc-tarballs/mingw/libgmp*.tar.gz      ||
-             test inplace/mingw -ot ghc-tarballs/mingw/libmpc*.tar.gz      ||
-             test inplace/mingw -ot ghc-tarballs/mingw/libmpfr*.tar.gz     ||
-             test inplace/mingw -ot ghc-tarballs/mingw/libstdc*.tar.lzma   ||
-             test inplace/mingw -ot ghc-tarballs/mingw/mingwrt*-dev.tar.gz ||
-             test inplace/mingw -ot ghc-tarballs/mingw/mingwrt*-dll.tar.gz ||
-             test inplace/mingw -ot ghc-tarballs/mingw/w32api*.tar.lzma
-        then
-            AC_MSG_NOTICE([Making in-tree mingw tree])
-            rm -rf inplace/mingw
-            mkdir inplace/mingw
-            (
-                cd inplace/mingw &&
-                tar --lzma -xf ../../ghc-tarballs/mingw/binutils*.tar.lzma  &&
-                tar --lzma -xf ../../ghc-tarballs/mingw/gcc-core*.tar.lzma  &&
-                tar --lzma -xf ../../ghc-tarballs/mingw/gcc-c++*.tar.lzma   &&
-                tar --lzma -xf ../../ghc-tarballs/mingw/libgcc*.tar.lzma    &&
-                tar --lzma -xf ../../ghc-tarballs/mingw/libgmp*.tar.lzma    &&
-                tar --lzma -xf ../../ghc-tarballs/mingw/libmpc*.tar.lzma    &&
-                tar --lzma -xf ../../ghc-tarballs/mingw/libmpfr*.tar.lzma   &&
-                tar --lzma -xf ../../ghc-tarballs/mingw/libstdc*.tar.lzma   &&
-                tar -z     -xf ../../ghc-tarballs/mingw/mingwrt*-dev.tar.gz &&
-                tar -z     -xf ../../ghc-tarballs/mingw/mingwrt*-dll.tar.gz &&
-                tar --lzma -xf ../../ghc-tarballs/mingw/w32api*.tar.lzma    &&
-                mv bin/gcc.exe bin/realgcc.exe
-            )
-            PATH=`pwd`/inplace/mingw/bin:$PATH inplace/mingw/bin/realgcc.exe driver/gcc/gcc.c driver/utils/cwrapper.c driver/utils/getLocation.c -Idriver/utils -o inplace/mingw/bin/gcc.exe
-            AC_MSG_NOTICE([In-tree mingw tree created])
-        fi
+        local file_md5sum="${file_md5sum_x86}"
     else
-        # NB. If you update the tarballs to a new version of gcc, don't
-        # forget to tweak the paths in driver/gcc/gcc.c.
-        if ! test -d inplace/mingw ||
-             test inplace/mingw -ot ghc-tarballs/mingw64/*.tar.bz2
-        then
-            AC_MSG_NOTICE([Making in-tree mingw tree])
-            rm -rf inplace/mingw
-            mkdir inplace/mingw
-            (
-                cd inplace/mingw &&
-                tar -jxf ../../ghc-tarballs/mingw64/*.tar.bz2
-            )
-            AC_MSG_NOTICE([In-tree mingw tree created])
-        fi
+        local file_md5sum="${file_md5sum_x64}"
     fi
 
+    local mingw_toolchain="$(basename $mingw_url)"
+    local mingw_w64="${tarball_dir}/${tarball_dest_dir}/${mingw_toolchain}"
+
+    download_file "${mingw_url}" "${file_md5sum}" "${mingw_w64}" "${mingw_toolchain}"
+
+    # Mark the tree as needing updates by deleting the folder
+    if test -d inplace/mingw && test inplace/mingw -ot "$mingw_w64"
+    then
+        AC_MSG_NOTICE([In-tree MinGW-w64 tree requires updates...])
+        rm -rf inplace/mingw
+    fi
+}
+
+set_up_tarballs() {
+    local mingw_base_url="http://repo.msys2.org/mingw"
+    local package_prefix="mingw-w64"
+
+    local format_url="${mingw_base_url}/${mingw_arch}/${package_prefix}-${mingw_arch}"
+
+    download_and_extract "${format_url}-crt-git-5.0.0.4531.49c7046-1-any.pkg.tar.xz"           "dd39323140c0c1b3e065e9edb1a66779" "ac22cedd38229bcd57f5999e4734054f"
+    download_and_extract "${format_url}-winpthreads-git-5.0.0.4538.78dca70-1-any.pkg.tar.xz"   "0b14fe27790e94db454fbb3564e79a73" "65cf07b6f42a1a62d1844e08190cab0d"
+    download_and_extract "${format_url}-headers-git-5.0.0.4531.49c7046-1-any.pkg.tar.xz"       "6ee9e3c2f9d3e507f60ee33d19417dc2" "f49a19cdea93998c33ac90ceb9570350"
+    download_and_extract "${format_url}-libwinpthread-git-5.0.0.4538.78dca70-1-any.pkg.tar.xz" "fbb2114aa7fbb5507e21d8a2ea265cfd" "31ed10e2d8891f6251d968f81bfdd274"
+    download_and_extract "${format_url}-zlib-1.2.8-8-any.pkg.tar.xz"                           "7f519cb6defa27a90c5353160cf088d4" "6a2f4a70ccb24acca70a01da331699a6"
+    download_and_extract "${format_url}-isl-0.14.1-2-any.pkg.tar.xz"                           "4cd20fe75ed9ef03e260d529042cb742" "dc0e0a7fd23a8193cccb0bf8d7267685"
+    download_and_extract "${format_url}-mpc-1.0.3-2-any.pkg.tar.xz"                            "719e76fa7a54a8676d2e60af3bb13c45" "df1a7d4050568d83c265ae78c32ef30b"
+    download_and_extract "${format_url}-mpfr-3.1.3.p0-2-any.pkg.tar.xz"                        "e9cbd2402ac1afe6e86c102223b90dcb" "6e3b9ec27edab394aa41536839afdafe"
+    download_and_extract "${format_url}-gmp-6.0.0-3-any.pkg.tar.xz"                            "c02f9759cd0140a6d8ea69ef5a88e167" "2970d4d8b176f8f36ae2d39269b25cce"
+    download_and_extract "${format_url}-gcc-libs-5.2.0-3-any.pkg.tar.xz"                       "a9bd2e65cb350cc8f8a6deb6d3b346a8" "9c2ed24989e14fdf0c548a5215374660"
+    download_and_extract "${format_url}-binutils-2.25.1-1-any.pkg.tar.xz"                      "997e9c2166fb851916cd8ac1bc9c6180" "7cb9f5f50a7103da41f7ec7547c09707"
+    download_and_extract "${format_url}-libiconv-1.14-5-any.pkg.tar.xz"                        "2c99a163689ba8257627bb07274b3f86" "37418c6be92ef20be17cdc9fe844af35"
+    download_and_extract "${format_url}-gcc-5.2.0-3-any.pkg.tar.xz"                            "efe6d6afc18aab89dc01e7ddcd2523a6" "0b697ce61112ba6e5a3c4d565957ea4e"
+
+    # Extract all the tarballs in one go
+    if ! test -d inplace/mingw
+    then
+        AC_MSG_NOTICE([Extracting Windows toolchain from archives (may take a while)...])
+        rm -rf inplace/mingw
+        local base_dir="../${tarball_dir}/${tarball_dest_dir}"
+        ( cd inplace &&
+        find "${base_dir}" -name "*.tar.xz" -exec tar xfJ {} \; &&
+        rm ".MTREE" &&
+        rm ".PKGINFO" &&
+        cd .. ) || fail "Error: Could not extract Windows toolchains."
+
+        mv "inplace/${tarball_mingw_dir}" inplace/mingw &&
+        touch inplace/mingw
+
+        # NB. Now since the GCC is hardcoded to use /mingw32 we need to
+        # make a wrapper around it to give it the proper paths
+        mv inplace/mingw/bin/gcc.exe inplace/mingw/bin/realgcc.exe
+        PATH=`pwd`/inplace/mingw/bin:$PATH
+        inplace/mingw/bin/realgcc.exe driver/gcc/gcc.c driver/utils/cwrapper.c driver/utils/getLocation.c -Idriver/utils -o inplace/mingw/bin/gcc.exe
+
+        AC_MSG_NOTICE([In-tree MingW-w64 tree created])
+    fi
+}
+
+if test "$HostOS" = "mingw32"
+then
+    test -d inplace || mkdir inplace
+
+    # NB. Download and extract the MingW-w64 distribution if required
+    set_up_tarballs
+
     mingwbin="$hardtop/inplace/mingw/bin/"
     CC="${mingwbin}gcc.exe"
     LD="${mingwbin}ld.exe"
@@ -384,6 +423,9 @@ then
     OBJDUMP="${mingwbin}objdump.exe"
     fp_prog_ar="${mingwbin}ar.exe"
 
+    # NB. Download the perl binaries if required
+    download_file "https://github.com/ghc/ghc-tarballs/blob/master/perl/ghc-perl-1.tar.gz?raw=true" "b21d1681b61cf7a024e854096285b02e" "ghc-tarballs/perl/ghc-perl-1.tar.gz" "Windows Perl binary distributions" "--insecure"
+
     if ! test -d inplace/perl ||
          test inplace/perl -ot ghc-tarballs/perl/ghc-perl*.tar.gz
     then