Bug 1488148 - Rework CI images, r=jcj

This does some fairly major restructuring of the docker images we use for CI.
The genesis of the change is that we were pulling a version of clang that didn't
work for fuzzing tests.  It turns out that is a use case that is not
well-supported anyway, and we have open bugs on it, but this installs
workarounds for all the problems I found.

Firstly, our images were bloated.  This slims down most of the images.  The
biggest gains are in the clang-format image (down to around a fifth of its
previous size).  The main linux image we use for building and running tests is
also less than half its original size.

To achieve that, I had to make two new images.  One for all the esoteric builds
we run (we compile with multiple gcc and clang versions, and I've added some
more to that list).  That's a fairly sizeable image.  The other is for the
interop and bogo suites, where we rely on having Rust and go available.  go adds
a tidy 250Mb to an image, and Rust adds 750Mb.  Using an image with both of
those for regular builds can't be good for performance.  I didn't expect to see
real performance gains here, but the Linux build (32-bit, default config) went
from 4:18 down to 2:42 (roughly).  The bulk of that time is accounted for by
downloading the docker image, so it's clear that an optimization worth spending
the time on.

Secondly, we had a lot of custom configuration stuff in the builds.  This
removes most of that in favour of using stock Ubuntu packages from 18.04.

The one exception here is - I hope - temporary.  As noted in the bug comments,
the current release of LLVM 6 has a bug where you can't run address sanitizer on
a 32-bit machine if it has glibc 2.27 (which Ubuntu 18.04 does).  That's fairly
crippling because we need a newer version of LLVM than runs by default on Ubuntu
16.04, so we're stuck with installing a non-stock version for 32-bit runs.  I've
opted to (temporarily) run 16.04 with an LLVM from the LLVM project.

The final change, which is minor, but a little odd and worth noting: the images
now rely on "localhost.localdomain" being aliased to the local machine.  This is
something :wcosta has done for us (thanks!).  Thus, we no longer have to run as
root so that we can tweak /etc/hosts before we run.  There is a little cleanup
related to this, but nothing significant.  (The scripts still include the `su
worker` tweak for aarch64, but I've added a guard and we can remove that with
bug 1488325.)

There is still more work to be done for the HACL* and SAW builds, which use some
very strange configurations.  Also, all of the aarch64 images aren't built
automatically, so we use images from Franziskus' dockerhub account.  This is not
good.  After digging around a little, there's probably something to be done with
QEMU, but I decided that was a project for another time.

2 files changed, 24 insertions, 59 deletions

diff --git a/automation/clang-format/Dockerfile b/automation/clang-format/Dockerfile
index 163c9b8fa..e74dac09f 100644
--- a/automation/clang-format/Dockerfile
+++ b/automation/clang-format/Dockerfile
@@ -1,26 +1,35 @@
-FROM ubuntu:16.04
-MAINTAINER Franziskus Kiefer <franziskuskiefer@gmail.com>
+# Minimal image with clang-format 3.9.
+FROM ubuntu:18.04
+LABEL maintainer="Martin Thomson <martin.thomson@gmail.com>"
 
-RUN useradd -d /home/worker -s /bin/bash -m worker
-WORKDIR /home/worker
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends \
+    ca-certificates \
+    clang-format-3.9 \
+    locales \
+    mercurial \
+ && rm -rf /var/lib/apt/lists/* \
+ && apt-get autoremove -y && apt-get clean -y
 
-# Install dependencies.
-ADD setup.sh /tmp/setup.sh
-RUN bash /tmp/setup.sh
+RUN update-alternatives --install /usr/bin/clang-format \
+    clang-format $(which clang-format-3.9) 10
 
-# Change user.
-USER worker
-
-# Env variables.
-ENV HOME /home/worker
 ENV SHELL /bin/bash
 ENV USER worker
-ENV LOGNAME worker
+ENV LOGNAME $USER
+ENV HOME /home/$USER
 ENV HOSTNAME taskcluster-worker
 ENV LANG en_US.UTF-8
-ENV LC_ALL en_US.UTF-8
+ENV LC_ALL $LANG
 ENV HOST localhost
 ENV DOMSUF localdomain
 
-# Entrypoint.
+RUN locale-gen $LANG \
+ && DEBIAN_FRONTEND=noninteractive dpkg-reconfigure locales
+
+RUN useradd -d $HOME -s $SHELL -m $USER
+WORKDIR $HOME
+USER $USER
+
+# Entrypoint - which only works if /home/worker/nss is mounted.
 ENTRYPOINT ["/home/worker/nss/automation/clang-format/run_clang_format.sh"]
diff --git a/automation/clang-format/setup.sh b/automation/clang-format/setup.sh
deleted file mode 100644
index beac9e905..000000000
--- a/automation/clang-format/setup.sh
+++ /dev/null
@@ -1,44 +0,0 @@
-#!/usr/bin/env bash
-
-set -v -e -x
-
-# Update packages.
-export DEBIAN_FRONTEND=noninteractive
-apt-get -y update && apt-get -y upgrade
-
-# Install packages.
-apt_packages=()
-apt_packages+=('ca-certificates')
-apt_packages+=('curl')
-apt_packages+=('xz-utils')
-apt_packages+=('mercurial')
-apt_packages+=('git')
-apt_packages+=('locales')
-apt-get install -y --no-install-recommends ${apt_packages[@]}
-
-# Download clang.
-curl -L https://releases.llvm.org/3.9.1/clang+llvm-3.9.1-x86_64-linux-gnu-ubuntu-16.04.tar.xz -o clang.tar.xz
-curl -L https://releases.llvm.org/3.9.1/clang+llvm-3.9.1-x86_64-linux-gnu-ubuntu-16.04.tar.xz.sig -o clang.tar.xz.sig
-# Verify the signature.
-gpg --keyserver pool.sks-keyservers.net --recv-keys B6C8F98282B944E3B0D5C2530FC3042E345AD05D
-gpg --verify clang.tar.xz.sig
-# Install into /usr/local/.
-tar xJvf *.tar.xz -C /usr/local --strip-components=1
-
-# Cleanup.
-function cleanup() {
-  rm -f clang.tar.xz clang.tar.xz.sig
-}
-trap cleanup ERR EXIT
-
-locale-gen en_US.UTF-8
-dpkg-reconfigure locales
-
-# Cleanup.
-rm -rf ~/.ccache ~/.cache
-apt-get autoremove -y
-apt-get clean
-apt-get autoclean
-
-# We're done. Remove this script.
-rm $0