ENH: Support character string arrays

TST: added test for issue #18684

ENH: f2py opens files with correct encoding, fixes #635

TST: added test for issue #6308

TST: added test for issue #4519

TST: added test for issue #3425

ENH: Implement user-defined hooks support for post-processing f2py data structure. Implement character BC hook.

ENH: Add support for detecting utf-16 and utf-32 encodings.

6 files changed, 91 insertions, 11 deletions

diff --git a/doc/source/f2py/advanced.rst b/doc/source/f2py/advanced.rst
index cf9984380..b310172e2 100644
--- a/doc/source/f2py/advanced.rst
+++ b/doc/source/f2py/advanced.rst
@@ -96,4 +96,54 @@ and the corresponding <C type>. The <C type> can be one of the following::
     complex_long_double
     string
 
-For more information, see the F2Py source code ``numpy/f2py/capi_maps.py``.
+For more information, see F2Py source code ``numpy/f2py/capi_maps.py``.
+
+.. _Character strings:
+
+Character strings
+=================
+
+Assumed length chararacter strings
+-----------------------------------
+
+In Fortran, assumed length character string arguments are declared as
+``character*(*)`` or ``character(len=*)``, that is, the length of such
+arguments are determined by the actual string arguments at runtime.
+For ``intent(in)`` arguments, this lack of length information poses no
+problems for f2py to construct functional wrapper functions. However,
+for ``intent(out)`` arguments, the lack of length information is
+problematic for f2py generated wrappers because there is no size
+information available for creating memory buffers for such arguments
+and F2PY assumes the length is 0.  Depending on how the length of
+assumed length character strings are specified, there exist ways to
+workaround this problem, as exemplified below.
+
+If the length of the ``character*(*)`` output argument is determined
+by the state of other input arguments, the required connection can be
+established in a signature file or within a f2py-comment by adding an
+extra declaration for the corresponding argument that specifies the
+length in character selector part. For example, consider a Fortran
+file ``asterisk1.f90``:
+
+.. include:: asterisk1.f90
+  :literal:
+
+Compile it with ``f2py -c asterisk1.f90 -m asterisk1`` and then in Python:
+
+.. include:: asterisk1_session.dat
+  :literal:
+
+Notice that the extra declaration ``character(f2py_len=12) s`` is
+interpreted only by f2py and in the ``f2py_len=`` specification one
+can use C-expressions as a length value.
+
+In the following example:
+
+.. include:: asterisk2.f90
+  :literal:
+
+the lenght of output assumed length string depends on an input
+argument ``n``, after wrapping with F2PY, in Python:
+
+.. include:: asterisk2_session.dat
+  :literal:
diff --git a/doc/source/f2py/asterisk1.f90 b/doc/source/f2py/asterisk1.f90
new file mode 100644
index 000000000..2631490d0
--- /dev/null
+++ b/doc/source/f2py/asterisk1.f90
@@ -0,0 +1,5 @@
+subroutine foo1(s)
+  character*(*), intent(out) :: s
+  !f2py character(f2py_len=12) s
+  s = "123456789A12"
+end subroutine foo1
diff --git a/doc/source/f2py/asterisk1_session.dat b/doc/source/f2py/asterisk1_session.dat
new file mode 100644
index 000000000..8aa9c5ba0
--- /dev/null
+++ b/doc/source/f2py/asterisk1_session.dat
@@ -0,0 +1,3 @@
+>>> import asterisk1
+>>> asterisk1.foo1()
+b'123456789A12'
diff --git a/doc/source/f2py/asterisk2.f90 b/doc/source/f2py/asterisk2.f90
new file mode 100644
index 000000000..278e02b4a
--- /dev/null
+++ b/doc/source/f2py/asterisk2.f90
@@ -0,0 +1,6 @@
+subroutine foo2(s, n)
+  character(len=*), intent(out) :: s
+  integer, intent(in) :: n
+  !f2py character(f2py_len=n), depend(n) :: s
+  s = "123456789A123456789B"(1:n)
+end subroutine foo2
diff --git a/doc/source/f2py/asterisk2_session.dat b/doc/source/f2py/asterisk2_session.dat
new file mode 100644
index 000000000..5fb8bef8b
--- /dev/null
+++ b/doc/source/f2py/asterisk2_session.dat
@@ -0,0 +1,6 @@
+>>> import asterisk
+>>> asterisk.foo2(2)
+b'12'
+>>> asterisk.foo2(12)
+b'123456789A12'
+>>> 
diff --git a/doc/source/f2py/signature-file.rst b/doc/source/f2py/signature-file.rst
index cea3682c2..bf358528f 100644
--- a/doc/source/f2py/signature-file.rst
+++ b/doc/source/f2py/signature-file.rst
@@ -645,21 +645,20 @@ A C expression may contain:
   according to given dependence relations;
 * the following CPP macros:
 
-  * ``rank(<name>)``
+  ``f2py_rank(<name>)``
     Returns the rank of an array ``<name>``.
-
-  * ``shape(<name>,<n>)``
+  ``f2py_shape(<name>, <n>)``
     Returns the ``<n>``-th dimension of an array ``<name>``.
-
-  * ``len(<name>)``
+  ``f2py_len(<name>)``
     Returns the length of an array ``<name>``.
-
-  * ``size(<name>)``
+  ``f2py_size(<name>)``
     Returns the size of an array ``<name>``.
-
-  * ``slen(<name>)``
+  ``f2py_itemsize(<name>)``
+    Returns the itemsize of an array ``<name>``.
+  ``f2py_slen(<name>)``
     Returns the length of a string ``<name>``.
 
+
 For initializing an array ``<array name>``, F2PY generates a loop over
 all indices and dimensions that executes the following
 pseudo-statement::
@@ -706,4 +705,15 @@ Currently, multi-line blocks can be used in the following constructs:
 
 * as a list of C arrays of the ``pymethoddef`` statement;
 
-* as a documentation string.
++ as documentation string.
+
+Extended char-selector
+-----------------------
+
+F2PY extends char-selector specification, usable within a signature
+file or a F2PY directive, as follows::
+
+  <extended-charselector> := <charselector>
+                          | (f2py_len= <len>)
+
+See :ref:`Character Strings` for usage.