gdb/python: fix FrameDecorator regression on Python 2

This commit:

  commit d1cab9876d72d867b2de82688f5f5a2a4b655edb
  Date:   Tue Sep 15 11:08:56 2020 -0600

      Don't use gdb_py_long_from_ulongest

Introduced a regression when GDB is compiled with Python 2.  The frame
filter API expects the gdb.FrameDecorator.function () method to return
either a string (the name of a function) or an address, which GDB then
uses to lookup a msymbol.

If the address returned from gdb.FrameDecorator.function () comes from
gdb.Frame.pc () then before the above commit we would always expect to
see a PyLong object.

After the above commit we might (on Python 2) get a PyInt object.

The GDB code does not expect to see a PyInt, and only checks for a
PyLong, we then see an error message like:

  RuntimeError: FrameDecorator.function: expecting a String, integer or None.

This commit just adds an additional call to PyInt_Check which handle
the missing case.

I had already written a test case to cover this issue before spotting
that the gdb.python/py-framefilter.exp test also triggers this
failure.  As the new test case is slightly different I have kept it
in.

The new test forces the behaviour of gdb.FrameDecorator.function
returning an address.  The reason the existing test case hits this is
due to the behaviour of the builtin gdb.FrameDecorator base class.  If
the base class behaviour ever changed then the return an address case
would only be tested by the new test case.

gdb/ChangeLog:

	* python/py-framefilter.c (py_print_frame): Use PyInt_Check as
	well as PyLong_Check for Python 2.

gdb/testsuite/ChangeLog:

	* gdb.python/py-framefilter-addr.c: New file.
	* gdb.python/py-framefilter-addr.exp: New file.
	* gdb.python/py-framefilter-addr.py: New file.

6 files changed, 169 insertions, 1 deletions

diff --git a/gdb/ChangeLog b/gdb/ChangeLog
index 0ac6b29443e..2e4628b87c5 100644
--- a/gdb/ChangeLog
+++ b/gdb/ChangeLog
@@ -1,3 +1,8 @@
+2021-03-16  Andrew Burgess  <andrew.burgess@embecosm.com>
+
+	* python/py-framefilter.c (py_print_frame): Use PyInt_Check as
+	well as PyLong_Check for Python 2.
+
 2021-03-15  Tom Tromey  <tromey@adacore.com>
 
 	PR build/27579:
diff --git a/gdb/python/py-framefilter.c b/gdb/python/py-framefilter.c
index 6dd741ab704..18071982e20 100644
--- a/gdb/python/py-framefilter.c
+++ b/gdb/python/py-framefilter.c
@@ -924,7 +924,11 @@ py_print_frame (PyObject *filter, frame_filter_flags flags,
 
 	      function = function_to_free.get ();
 	    }
-	  else if (PyLong_Check (py_func.get ()))
+	  else if (PyLong_Check (py_func.get ())
+#if PY_MAJOR_VERSION == 2
+		   || PyInt_Check (py_func.get ())
+#endif
+		   )
 	    {
 	      CORE_ADDR addr;
 	      struct bound_minimal_symbol msymbol;
diff --git a/gdb/testsuite/ChangeLog b/gdb/testsuite/ChangeLog
index 374ea394172..1e23b91d9a2 100644
--- a/gdb/testsuite/ChangeLog
+++ b/gdb/testsuite/ChangeLog
@@ -1,5 +1,11 @@
 2021-03-16  Andrew Burgess  <andrew.burgess@embecosm.com>
 
+	* gdb.python/py-framefilter-addr.c: New file.
+	* gdb.python/py-framefilter-addr.exp: New file.
+	* gdb.python/py-framefilter-addr.py: New file.
+
+2021-03-16  Andrew Burgess  <andrew.burgess@embecosm.com>
+
 	* gdb.threads/execl.exp: Remove duplicate 'info threads' test.
 	Make use of $gdb_test_name instead of creating a separate $test
 	variable.
diff --git a/gdb/testsuite/gdb.python/py-framefilter-addr.c b/gdb/testsuite/gdb.python/py-framefilter-addr.c
new file mode 100644
index 00000000000..446ec3ed9b9
--- /dev/null
+++ b/gdb/testsuite/gdb.python/py-framefilter-addr.c
@@ -0,0 +1,40 @@
+/* This testcase is part of GDB, the GNU debugger.
+
+   Copyright 2021 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 3 of the License, or
+   (at your option) any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
+
+int
+func3 ()
+{
+  return 0;	/* Break here.  */
+}
+
+int
+func2 ()
+{
+  return func3 ();
+}
+
+int
+func1 ()
+{
+  return func2 ();
+}
+
+int
+main ()
+{
+  return func1 ();
+}
diff --git a/gdb/testsuite/gdb.python/py-framefilter-addr.exp b/gdb/testsuite/gdb.python/py-framefilter-addr.exp
new file mode 100644
index 00000000000..108509cfcdd
--- /dev/null
+++ b/gdb/testsuite/gdb.python/py-framefilter-addr.exp
@@ -0,0 +1,61 @@
+# Copyright (C) 2021 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# This file is part of the GDB testsuite.  It tests Python-based
+# frame-filters when the 'function ()' method on FrameDecorator
+# returns the address for the function being decorated.
+
+load_lib gdb-python.exp
+
+standard_testfile
+
+if { [prepare_for_testing "failed to prepare" ${testfile} ${srcfile}] } {
+    return -1
+}
+
+# Skip all tests if Python scripting is not enabled.
+if { [skip_python_tests] } { continue }
+
+if ![runto_main] {
+   return -1
+}
+
+# Run to our test breakpoint.
+gdb_breakpoint [gdb_get_line_number "Break here"]
+gdb_continue_to_breakpoint "run to test breakpoint"
+
+gdb_test "bt" \
+    [multi_line \
+	 "#0  func3 \\(\\) at \[^\r\n\]+/py-framefilter-addr.c:21" \
+	 "#1  $hex in func2 \\(\\) at \[^\r\n\]+/py-framefilter-addr.c:27" \
+	 "#2  $hex in func1 \\(\\) at \[^\r\n\]+/py-framefilter-addr.c:33" \
+	 "#3  $hex in main \\(\\) at \[^\r\n\]+/py-framefilter-addr.c:39" ] \
+    "backtrace without frame filters"
+
+# Make the frame filters Python script available.
+set remote_python_file \
+    [gdb_remote_download host ${srcdir}/${subdir}/${testfile}.py \
+	 ${testfile}.py]
+
+# And load it into GDB.
+gdb_test_no_output "source ${remote_python_file}" "load python file"
+
+gdb_test "bt" \
+    [multi_line \
+	 "#0  func3 \\(\\) at \[^\r\n\]+/py-framefilter-addr.c:21" \
+	 "#1  $hex in func2 \\(\\) at \[^\r\n\]+/py-framefilter-addr.c:27" \
+	 "#2  $hex in func1 \\(\\) at \[^\r\n\]+/py-framefilter-addr.c:33" \
+	 "#3  $hex in main \\(\\) at \[^\r\n\]+/py-framefilter-addr.c:39" ] \
+    "backtrace with frame filters"
diff --git a/gdb/testsuite/gdb.python/py-framefilter-addr.py b/gdb/testsuite/gdb.python/py-framefilter-addr.py
new file mode 100644
index 00000000000..a27879a4760
--- /dev/null
+++ b/gdb/testsuite/gdb.python/py-framefilter-addr.py
@@ -0,0 +1,52 @@
+# Copyright (C) 2021 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import gdb
+import itertools
+from gdb.FrameDecorator import FrameDecorator
+import copy
+
+# A FrameDecorator that just returns gdb.Frame.pc () from 'function'.
+# We want to ensure that GDB correctly handles this case.
+class Function_Returns_Address (FrameDecorator):
+
+    def __init__(self, fobj):
+        super (Function_Returns_Address, self).__init__ (fobj)
+        self._fobj = fobj
+
+    def function (self):
+        frame = self.inferior_frame ()
+        return frame.pc ()
+
+class Frame_Filter ():
+
+    def __init__ (self):
+        self.name = "function_returns_address"
+        self.priority = 100
+        self.enabled = True
+        gdb.frame_filters [self.name] = self
+
+    def filter (self, frame_iter):
+        # Python 3.x moved the itertools.imap functionality to map(),
+        # so check if it is available.
+        if hasattr(itertools, "imap"):
+            frame_iter = itertools.imap (Function_Returns_Address,
+                                         frame_iter)
+        else:
+            frame_iter = map(Function_Returns_Address, frame_iter)
+
+        return frame_iter
+
+Frame_Filter()