Use new is_utf8_valid_partial_char()

This new function can be used in the implementation of the file test
operators, -B and -T, to see if the whole fixed length buffer is valid
UTF-8.  Previously if all bytes were UTF-8 except the bytes at the end
that could have been a partial character, it assumed the whole thing was
UTF-8.  This improves the prediction slightly

1 files changed, 4 insertions, 4 deletions

diff --git a/pp_sys.c b/pp_sys.c
index 7e13e38dac..a198d4e525 100644
--- a/pp_sys.c
+++ b/pp_sys.c
@@ -3559,10 +3559,10 @@ PP(pp_fttext)
         const U8 *ep;
 
         /* Here contains a variant under UTF-8 .  See if the entire string is
-         * UTF-8.  But the buffer may end in a partial character, so consider
-         * it UTF-8 if the first non-UTF8 char is an ending partial */
-        if (is_utf8_string_loc((U8 *) s, len, &ep)
-            || ep + UTF8SKIP(ep)  > (U8 *) (s + len))
+         * UTF-8.  But the buffer may end in a partial character, so if it
+         * failed, see if the failure was due just to that */
+        if (   is_utf8_string_loc((U8 *) s, len, &ep)
+            || is_utf8_valid_partial_char(ep, (U8 *) s + len))
         {
             if (PL_op->op_type == OP_FTTEXT) {
                 FT_RETURNYES;