configure: Don't declare functions in AC_TRY_COMPILE and AC_LANG_PROGRAM

Both AC_TRY_COMPILE and AC_LANG_PROGRAM put code passed to their second
arguments to the body of the main function. This means that we cannot
and should not declare functions there, or we end up checking whether
the compiler support nested functions instead of whether a compiler or
linker flag is supported.

GCC supports nested functions and tests succeed. Clang doesn't support
nested functions, so tests fail and -fvisibility=hidden won't be used.
This means that functions which are not intended to be used by other
programs, such as gtk_menu_tracker*, gtk_action_observ*,
gtk_menu_muxer_*, become global symbols with default visibility.

GNOME Shell has a private library libgnome-shell-menu.so, which also has
symbols gtk_menu_tracker*, gtk_action_observ*, gtk_menu_muxer_* that are
intended to be used by GNOME Shell itself. When GNOME Shell still used
Autotools build system, the executable gnome-shell explicitly linked to
libgnome-shell-menu.so, so the linker loaded libgnome-shell-menu.so
before libgtk-3.so.0 and GNOME Shell used correct symbols from its
private library.

However, after GNOME Shell switched to Meson build system, gnome-shell
executable no longer lists libgnome-shell-menu.so as its dependency.
Even if we adds it to the build file, it won't be listed in DT_NEEDED of
gnome-shell because Meson uses -Wl,--as-needed by default. This causes
the runtime linker to load libgtk-3.so.0 before libgnome-shell-menu.so
and symbols gtk_menu_tracker*, gtk_action_observ*, gtk_menu_muxer_* are
bound to libgtk-3.so.0 instead of libgnome-shell-menu.so. GNOME Shell
hangs when opening more than one window because it uses functions from
the wrong library.

This problem is already fixed in OpenBSD ports. The article describing
it can be found on OpenBSD Journal with this link:
https://undeadly.org/cgi?action=article;sid=20170930133438

https://bugzilla.gnome.org/show_bug.cgi?id=791943

1 files changed, 2 insertions, 3 deletions

diff --git a/configure.ac b/configure.ac
index 904b2ba50b..16e011d5fa 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1820,7 +1820,7 @@ case "$host" in
     SAVED_CFLAGS="${CFLAGS}"
     CFLAGS="-fvisibility=hidden"
     AC_MSG_CHECKING([for -fvisibility=hidden compiler flag])
-    AC_TRY_COMPILE([], [int main (void) { return 0; }],
+    AC_TRY_COMPILE([], [return 0],
                    AC_MSG_RESULT(yes)
                    enable_fvisibility_hidden=yes,
                    AC_MSG_RESULT(no)
@@ -1901,8 +1901,7 @@ AC_ARG_ENABLE(Bsymbolic,
               [SAVED_LDFLAGS="${LDFLAGS}"
                AC_MSG_CHECKING([for -Bsymbolic-functions linker flag])
                LDFLAGS=-Wl,-Bsymbolic-functions
-               AC_LINK_IFELSE([AC_LANG_PROGRAM([[]],
-                                               [[int main (void) { return 0; }]])],
+               AC_LINK_IFELSE([AC_LANG_PROGRAM([[]], [[return 0]])],
                               [AC_MSG_RESULT(yes)
                                enable_Bsymbolic=yes],
                               [AC_MSG_RESULT(no)