Merge remote-tracking branch 'fribidi/master' into unicode-10

12 files changed, 504 insertions, 400 deletions

diff --git a/HACKING b/HACKING
deleted file mode 100644
index e3f8d9a..0000000
--- a/HACKING
+++ /dev/null
@@ -1,2 +0,0 @@
-So you are interested in hacking GNU FriBidi?  Great...
-Join the mailing list and lets talk.
diff --git a/Makefile.am b/Makefile.am
index 66f4a4e..58f6e1b 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -11,7 +11,7 @@
 ACLOCAL_AMFLAGS = -I m4
 
 ## The order of subdirs is important, don't change without a reason.
-SUBDIRS = gen.tab charset lib bin doc test
+SUBDIRS = gen.tab charset lib bin doc test fribidi-vs-unicode
 
 EXTRA_DIST = bootstrap ChangeLog.old
 
diff --git a/README b/README
index dd08b47..42061c0 100644..120000
--- a/README
+++ b/README
@@ -1,153 +1 @@
-This is GNU FriBidi
-The Free Implementation of the Unicode Bidirectional Algorithm.
-
-Background
-==========
-
-One of the missing links stopping the penetration of free software in Middle
-East is the lack of support for the Arabic and Hebrew alphabets. In order to
-have proper Arabic and Hebrew support, the BiDi algorithm should have been
-implemented. It is our hope that this library will stimulate more free
-software in the Middle Eastern countries.
-
-Audience
-========
-
-It is our hope that this library will stimulate the implementation of Hebrew
-and Arabic in lots of free software. Here is a small list of projects that
-would benefit from the use of the GNU FriBidi library, but of course there are
-many more: Wine, Mozilla, Qt, KDE, lynx, OpenOffice.
-
-GNU FriBidi is already being used in projects like Pango (resulting in GTK+
-and GNOME using GNU FriBidi), AbiWord, MLTerm, MPlayer, and BiCon.
-
-Dependencies
-============
-
-Currently GNU FriBidi does not depend on any other library, but uses Glib if
-available.  It uses GNU Build System for build and installation on POSIX
-systems.
-
-Downloading
-===========
-
-The latest version of GNU FriBidi may be found at:
-
-   http://fribidi.org/
-
-Building
-========
-
-See INSTALL for a description of how to build and install GNU FriBidi.
-
-License
-=========
-
-GNU FriBidi is free software; you can redistribute it and/or
-modify it under the terms of the GNU Lesser General Public License
-as published by the Free Software Foundation; either version 2.1
-of the License, or (at your option) any later version.
-
-GNU FriBidi is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU Lesser General Public License for more details.
-
-You should have received a copy of the GNU Lesser General Public License
-along with GNU FriBidi, in a file named COPYING; if not, write to the
-Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
-Boston, MA  02110-1301  USA
-
-For licensing issues, contact <fribidi.license@gmail.com>.
-
-Implementation
-==============
-
-The library implements the algorithm described in the "Unicode Standard
-Annex #9, The Bidirectional Algorithm", available at
-http://www.unicode.org/unicode/reports/tr9/. GNU Fribidi has been tested
-exhaustively against the Unicode Reference Code, and to the best of our
-knowledge, it completely conforms to the specification, always producing
-the same result as the Reference Code.
-
-The library uses Unicode (UTF32) entirely. The character properties are
-automatically extracted from the Unicode data files, available from:
-
-	http://www.unicode.org/Public/UNIDATA/
-
-This means that every Unicode character is treated in strict accordance
-with the Unicode specification.
-
-There is a limited support for character set conversion from/to the UTF32
-encoding.  Data in these character sets must be converted into UTF32 before
-the library may be used.  iconv(3) can always do a better job on that, so you
-may find that the character sets conversion code is typically turned off on
-POSIX machines.
-
-The reordering of characters is typically done through the function:
-      
-     fribidi_boolean
-     fribidi_log2vis(/* input */
-		     FriBidiChar *str,
-		     FriBidiStrIndex len,
-		     FriBidiCharType *pbase_dir,
-		     /* output */
-		     FriBidiChar *visual_str,
-		     FriBidiStrIndex *position_L_to_V_list,
-		     FriBidiStrIndex *position_V_to_L_list,
-		     FriBidiLevel    *embedding_level_list
-		     )
-    
-
-where
-     str                    is the Unicode input string
-     len                    is the length of the unicode string
-     pbase_dir              is the input and output base direction. If 
-                            base == FRIBIDI_TYPE_ON then fribidi_log2vis 
-                            calculates the base direction on its own
-                            according to the BiDi algorithm.
-     visual_str             The reordered output unicode string.
-     position_L_to_V_list   Maps the positions in the logical string to 
-                            positions in the visual string.
-     position_V_to_L_list   Maps the positions in the visual string to 
-                            the positions in the logical string.
-     embedding_level_list   Returns the classification of each character. Even
-                            levels indicate LTR characters, and odd levels
-                            indicate RTL characters. The main use of this
-                            list is in interactive applications when the
-                            embedding e.g. level determines cursor display.
-
-In any of the output pointers == NULL, then that information is not 
-calculated.
-
-How it looks like
-=================
-
-Have a look at tests directory, to see some input and outputs, which
-CapRTL charset means that CAPITAL letters are right to left, and digits
-6, 7, 8, 9 are Arabic digits, try 'fribidi --charsetdesc CapRTL' for the
-full description.
-
-Executable
-==========
-
-There is also a command-line utilitity called fribidi that loops over
-the text of a file and performs the BiDi algorithm on each line, also
-used for testing the algorithm. Run fribidi with the --help option to
-learn about usage.  The command-line utility is known to have problems
-with line-breaking and ltov/vtol lists.
-
-Bugs and comments
-=================
-
-Report GNU FriBidi bugs at:
-
-	http://fribidi.org/bug
-
-And send your comments to:
-
-	fribidi@freedesktop.org
-
-
-Behdad Esfahbod
-behdad@gnu.org
+README.md
\ No newline at end of file
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..faa53fe
--- /dev/null
+++ b/README.md
@@ -0,0 +1,174 @@
+# GNU FriBidi
+
+The Free Implementation of the [Unicode Bidirectional Algorithm].
+
+
+## Background
+
+One of the missing links stopping the penetration of free software in Middle
+East is the lack of support for the Arabic and Hebrew alphabets. In order to
+have proper Arabic and Hebrew support, the bidi algorithm needs to be implemented. It is
+our hope that this library will stimulate more free software in the Middle
+Eastern countries.
+
+See [`HISTORY`](./HISTORY) on how the project started and evolved.
+
+
+## Audience
+
+It is our hope that this library will stimulate the implementation of Hebrew and
+Arabic support in lots of Free Software. Here is a small list of projects that
+would benefit from the use of the GNU FriBidi library, but of course there are
+many more: Wine, Mozilla, Qt, KDE, lynx, OpenOffice.
+
+GNU FriBidi is already being used in projects like Pango (resulting in [GTK+] and
+[GNOME] using GNU FriBidi), AbiWord, MLTerm, MPlayer, and BiCon.
+
+See [`USERS`](./USERS) for a list of projects using GNU FriBidi.
+
+
+## Dependencies
+
+Currently GNU FriBidi does not depend on any other library, but uses Glib if
+available. It uses GNU Build System for build and installation on POSIX systems.
+
+
+## Downloading
+
+The latest version of GNU FriBidi may be found at:
+<https://github.com/fribidi/fribidi>
+
+
+## Building
+
+Start with running the [`bootstrap`](./bootstrap) script and follow the
+isntructions.
+
+
+## License
+
+GNU FriBidi is Free Software; you can redistribute it and/or modify it under the
+terms of the [GNU Lesser General Public License] as published by the Free Software
+Foundation; either version 2.1 of the License, or (at your option) any later
+version.
+
+GNU FriBidi is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
+PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.
+
+You should have received a copy of the GNU Lesser General Public License along
+with GNU FriBidi, in a file named COPYING; if not, write to the Free Software
+Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+For licensing issues, contact <license@farsiweb.info>.
+
+
+## Implementation
+
+The library implements the algorithm described in the "Unicode Standard Annex
+\#9, The Bidirectional Algorithm", available at
+<http://www.unicode.org/unicode/reports/tr9/>.
+
+The library uses Unicode (UTF-32) entirely. The character properties are
+automatically extracted from the Unicode data files, available from
+<http://www.unicode.org/Public/UNIDATA/>. This means that every Unicode
+character is treated in strict accordance with the Unicode specification.
+
+There is a limited support for character set conversion from/to the UTF-32
+encoding. Data in these character sets must be converted into UTF-32 before the
+library may be used. iconv(3) can always do a better job on that, so you may
+find that the character sets conversion code is typically turned off on POSIX
+machines.
+
+
+### Conformance Status
+
+GNU Fribidi has been tested exhaustively against the [Unicode Reference Code],
+and to the best of our knowledge, it completely conforms to the specification,
+always producing the same result as the Reference Code, except for [Mirroring]
+additions, introduced in Unicode 6.3.0 of the specification.
+
+
+### API
+
+The reordering of characters is typically done through the function:
+
+```c
+fribidi_boolean fribidi_log2vis(
+    /* input */
+    FriBidiChar     *str,
+    FriBidiStrIndex len,
+    FriBidiCharType *pbase_dir,
+    /* output */
+    FriBidiChar     *visual_str,
+    FriBidiStrIndex *position_L_to_V_list,
+    FriBidiStrIndex *position_V_to_L_list,
+    FriBidiLevel    *embedding_level_list
+)
+```
+
+Where...
+
+* `str` is the Unicode input string.
+* `len` is the length of the Unicode string (`str`).
+* `pbase_dir` is the input and output base direction. If `pbase_dir ==
+  FRIBIDI_TYPE_ON` then `fribidi_log2vis()` calculates the base direction on its
+  own, according to the bidi algorithm.
+* `visual_str` is the reordered output unicode string.
+* `position_L_to_V_list` maps the positions in the logical string to positions
+  in the visual string.
+* `position_V_to_L_list` maps the positions in the visual string to the
+  positions in the logical string.
+* `embedding_level_list` returns the classification of each character. Here,
+  even numerical levels indicate LTR characters, and odd levels indicate RTL
+  characters. The main use of this list is in interactive applications where,
+  for example, the embedding level determines cursor display.
+
+If any of the output pointers is equal to `NULL`, then that information is not
+calculated.
+
+
+## How it looks like
+
+Have a look at [`test/`](./test) directory, to see some input and outputs.
+
+The `CapRTL` charset means that CAPITAL letters are right to left, and digits
+6, 7, 8, 9 are Arabic digits, try 'fribidi --charsetdesc CapRTL' for the full
+description.
+
+
+## Executable
+
+There is also a command-line utilitity called `fribidi` that loops over the text
+of a file and performs the bidi algorithm on each line, also used for testing
+the algorithm.
+
+Run `fribidi --help` to learn about usage.
+
+The command-line utility is known to have problems with line-breaking and
+logical-to-vertical/vertical-to-logical lists.
+
+
+## Bug Reports and Feedback
+
+Report bugs and general feedback at: <https://github.com/fribidi/fribidi/issues>
+
+The mailing list is the place for additional technical discussions and user
+questions: <https://lists.freedesktop.org/mailman/listinfo/fribidi>
+
+
+## Maintainers and Contributors
+
+* Dov Grobgeld <dov.grobgeld@gmail.com>
+* Behdad Esfahbod <behdad@gnu.org>
+
+See also [`AUTHORS`](./AUTHORS) and [`THANKS`](./THANKS) for the complete list
+of contributors.
+
+
+[Unicode Bidirectional Algorithm]: https://www.unicode.org/reports/tr9/
+[Unicode Reference Code]: https://www.unicode.org/reports/tr9/#Reference_Code
+[Mirroring]: https://www.unicode.org/reports/tr9/#Mirroring
+[GTK+]: https://www.gtk.org/
+[GNOME]: https://www.gnome.org/
+[GNU Lesser General Public License]: https://www.gnu.org/licenses/old-licenses/lgpl-2.1.en.html
diff --git a/bin/fribidi-main.c b/bin/fribidi-main.c
index cd2fb93..af9a99e 100644
--- a/bin/fribidi-main.c
+++ b/bin/fribidi-main.c
@@ -508,7 +508,7 @@ main (
 			      while (wid > 0 && idx < len)
 				{
 				  wid -=
-				    FRIBIDI_IS_EXPLICIT_OR_BN_OR_NSM
+				    FRIBIDI_IS_EXPLICIT_OR_ISOLATE_OR_BN_OR_NSM
 				    (fribidi_get_bidi_type (visual[idx])) ? 0
 				    : 1;
 				  idx++;
diff --git a/charset/fribidi-char-sets.c b/charset/fribidi-char-sets.c
index b62a079..f7156b4 100644
--- a/charset/fribidi-char-sets.c
+++ b/charset/fribidi-char-sets.c
@@ -114,7 +114,7 @@ static FriBidiCharSetHandler char_sets[FRIBIDI_CHAR_SETS_NUM + 1] = {
 };
 
 #if FRIBIDI_USE_GLIB+0
-# include <glib.h>
+# include <glib/gstrfuncs.h>
 # define fribidi_strcasecmp g_ascii_strcasecmp
 #else /* !FRIBIDI_USE_GLIB */
 static char
diff --git a/configure.ac b/configure.ac
index cc76f03..c98380c 100644
--- a/configure.ac
+++ b/configure.ac
@@ -155,9 +155,6 @@ else
 fi
 AC_SUBST(FRIBIDI_NO_DEPRECATED)
 
-AC_DEFINE(USE_SIMPLE_MALLOC,1,
-	  [Define to 1 if you want to use simple mallocs instead of memory chunks])
-
 # --disable-charsets
 AC_ARG_ENABLE(charsets,
               AC_HELP_STRING([--disable-charsets],
@@ -214,5 +211,5 @@ AC_CONFIG_FILES([fribidi.pc
 		 bin/Makefile
 		 doc/Makefile
 		 test/Makefile
-                 fribidi-vs-unicode/Makefile])
+		 fribidi-vs-unicode/Makefile])
 AC_OUTPUT
diff --git a/gen.tab/gen-joining-type-tab.c b/gen.tab/gen-joining-type-tab.c
index 3d2a7f4..8a3f024 100644
--- a/gen.tab/gen-joining-type-tab.c
+++ b/gen.tab/gen-joining-type-tab.c
@@ -151,6 +151,10 @@ static const char *ignored_bidi_types[] = {
   "LRO",
   "RLO",
   "PDF",
+  "LRI",
+  "RLI",
+  "FSI",
+  "PDI",
   NULL
 };
 
diff --git a/lib/fribidi-mem.c b/lib/fribidi-mem.c
deleted file mode 100644
index 5b1e958..0000000
--- a/lib/fribidi-mem.c
+++ /dev/null
@@ -1,140 +0,0 @@
-/* FriBidi
- * fribidi-mem.c - memory manipulation routines
- *
- * $Id: fribidi-mem.c,v 1.8 2006-01-31 03:23:13 behdad Exp $
- * $Author: behdad $
- * $Date: 2006-01-31 03:23:13 $
- * $Revision: 1.8 $
- * $Source: /home/behdad/src/fdo/fribidi/togit/git/../fribidi/fribidi2/lib/fribidi-mem.c,v $
- *
- * Authors:
- *   Behdad Esfahbod, 2001, 2002, 2004
- *
- * Copyright (C) 2004 Sharif FarsiWeb, Inc
- * Copyright (C) 2001,2002 Behdad Esfahbod
- * 
- * This library is free software; you can redistribute it and/or
- * modify it under the terms of the GNU Lesser General Public
- * License as published by the Free Software Foundation; either
- * version 2.1 of the License, or (at your option) any later version.
- * 
- * This library is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
- * Lesser General Public License for more details.
- * 
- * You should have received a copy of the GNU Lesser General Public License
- * along with this library, in a file named COPYING; if not, write to the
- * Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
- * Boston, MA 02110-1301, USA
- * 
- * For licensing issues, contact <fribidi.license@gmail.com>.
- */
-
-#include "common.h"
-
-#include "mem.h"
-
-#if FRIBIDI_USE_GLIB+0
-#else
-#if USE_SIMPLE_MALLOC+0
-#else
-
-struct _FriBidiMemChunk
-{
-  int atom_size;
-  int area_size;
-  int empty_size;
-  void *chunk;
-};
-
-FriBidiMemChunk *
-fribidi_mem_chunk_new (
-  /* input */
-  const char *name,
-  int atom_size,
-  unsigned long area_size,
-  int alloc_type
-)
-{
-  register FriBidiMemChunk *m;
-
-  fribidi_assert (area_size >= atom_size * 8);
-
-  m = (FriBidiMemChunk *) fribidi_malloc (sizeof (FriBidiMemChunk));
-  if LIKELY
-    (m)
-    {
-      m->atom_size = atom_size;
-      m->area_size = area_size;
-      m->empty_size = 0;
-      m->chunk = NULL;
-    }
-
-  return m;
-}
-
-void *
-fribidi_mem_chunk_alloc (
-  /* input */
-  FriBidiMemChunk *mem_chunk
-)
-{
-  fribidi_assert (mem_chunk);
-
-  if UNLIKELY
-    (mem_chunk->empty_size < mem_chunk->atom_size)
-    {
-      register void *chunk = fribidi_malloc (mem_chunk->area_size);
-      if LIKELY
-	(chunk)
-	{
-	  if (mem_chunk->chunk)
-	    *(void **) chunk =
-	      (char *) mem_chunk->chunk + mem_chunk->empty_size -
-	      mem_chunk->area_size;
-	  chunk = (char *) chunk + mem_chunk->atom_size;
-	  mem_chunk->chunk = chunk;
-	  mem_chunk->empty_size = mem_chunk->area_size - mem_chunk->atom_size;
-	}
-      else
-	return NULL;
-    }
-
-  {
-    register void *m = mem_chunk->chunk;
-    mem_chunk->chunk = (char *) mem_chunk->chunk + mem_chunk->atom_size;
-    mem_chunk->empty_size -= mem_chunk->atom_size;
-
-    return m;
-  }
-}
-
-void
-fribidi_mem_chunk_destroy (
-  /* input */
-  FriBidiMemChunk *mem_chunk
-)
-{
-  register void *chunk;
-
-  fribidi_assert (mem_chunk);
-
-  chunk =
-    (char *) mem_chunk->chunk + mem_chunk->empty_size - mem_chunk->area_size;
-  while LIKELY
-    (chunk)
-    {
-      register void *tofree = chunk;
-      chunk = *(void **) chunk;
-      fribidi_free (tofree);
-    }
-  fribidi_free (mem_chunk);
-}
-
-#endif /* !USE_SIMPLE_MALLOC */
-#endif /* !FRIBIDI_USE_GLIB */
-
-/* Editor directions:
- * vim:textwidth=78:tabstop=8:shiftwidth=2:autoindent:cindent
- */
diff --git a/lib/mem.h b/lib/mem.h
deleted file mode 100644
index 059c0a2..0000000
--- a/lib/mem.h
+++ /dev/null
@@ -1,97 +0,0 @@
-/* FriBidi
- * mem.h - memory manipulation routines
- *
- * $Id: mem.h,v 1.7 2006-01-31 03:23:13 behdad Exp $
- * $Author: behdad $
- * $Date: 2006-01-31 03:23:13 $
- * $Revision: 1.7 $
- * $Source: /home/behdad/src/fdo/fribidi/togit/git/../fribidi/fribidi2/lib/mem.h,v $
- *
- * Author:
- *   Behdad Esfahbod, 2001, 2002, 2004
- *
- * Copyright (C) 2004 Sharif FarsiWeb, Inc
- * Copyright (C) 2001,2002 Behdad Esfahbod
- * 
- * This library is free software; you can redistribute it and/or
- * modify it under the terms of the GNU Lesser General Public
- * License as published by the Free Software Foundation; either
- * version 2.1 of the License, or (at your option) any later version.
- * 
- * This library is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
- * Lesser General Public License for more details.
- * 
- * You should have received a copy of the GNU Lesser General Public License
- * along with this library, in a file named COPYING; if not, write to the
- * Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
- * Boston, MA 02110-1301, USA
- * 
- * For licensing issues, contact <fribidi.license@gmail.com>.
- */
-#ifndef _MEM_H
-#define _MEM_H
-
-#include "common.h"
-
-#include <fribidi-types.h>
-
-#include <fribidi-begindecls.h>
-
-#if FRIBIDI_USE_GLIB+0
-
-#ifndef __FRIBIDI_DOC
-# include <glib.h>
-#endif /* !__FRIBIDI_DOC */
-
-#define FriBidiMemChunk GMemChunk
-#define FRIBIDI_ALLOC_ONLY G_ALLOC_ONLY
-#define fribidi_mem_chunk_new g_mem_chunk_new
-#define fribidi_mem_chunk_alloc g_mem_chunk_alloc
-#define fribidi_mem_chunk_destroy g_mem_chunk_destroy
-
-#else /* !FRIBIDI_USE_GLIB */
-
-typedef struct _FriBidiMemChunk FriBidiMemChunk;
-
-#define FRIBIDI_ALLOC_ONLY      1
-
-#define fribidi_mem_chunk_new FRIBIDI_PRIVATESPACE(mem_chunk_new)
-FriBidiMemChunk *
-fribidi_mem_chunk_new (
-  const char *name,
-  int atom_size,
-  unsigned long area_size,
-  int alloc_type
-)
-     FRIBIDI_GNUC_HIDDEN FRIBIDI_GNUC_MALLOC FRIBIDI_GNUC_WARN_UNUSED;
-
-#define fribidi_mem_chunk_alloc FRIBIDI_PRIVATESPACE(mem_chunk_alloc)
-     void *fribidi_mem_chunk_alloc (
-  FriBidiMemChunk *mem_chunk
-)
-     FRIBIDI_GNUC_HIDDEN FRIBIDI_GNUC_MALLOC FRIBIDI_GNUC_WARN_UNUSED;
-
-#define fribidi_mem_chunk_destroy FRIBIDI_PRIVATESPACE(mem_chunk_destroy)
-     void fribidi_mem_chunk_destroy (
-  FriBidiMemChunk *mem_chunk
-) FRIBIDI_GNUC_HIDDEN;
-
-#endif /* !FRIBIDI_USE_GLIB */
-
-#define fribidi_chunk_new(type, chunk)        ( \
-		(type *) fribidi_mem_chunk_alloc (chunk) \
-	)
-
-#define fribidi_chunk_new_for_type(type) ( \
-		fribidi_mem_chunk_new(FRIBIDI, sizeof (type), \
-				FRIBIDI_CHUNK_SIZE, FRIBIDI_ALLOC_ONLY) \
-	)
-
-#include <fribidi-enddecls.h>
-
-#endif /* !_MEM_H */
-/* Editor directions:
- * vim:textwidth=78:tabstop=8:shiftwidth=2:autoindent:cindent
- */
diff --git a/notes.org b/notes.org
new file mode 100644
index 0000000..7a44ef6
--- /dev/null
+++ b/notes.org
@@ -0,0 +1,318 @@
+#+STARTUP: hidestars showall
+* <2014-03-08 Sat>
+** Running a test of the new features
+#+BEGIN_SRC sh
+cd test
+../bin/fribidi --charset CapRTL test_CapRTL_isolate.input 
+#+END_SRC
+   - No link to gen-bidi-type-tab .
+* <2014-03-16 Sun>
+** Running more extensive tests
+#+BEGIN_SRC sh
+../bin/fribidi --basedir --showinput --levels --charset CapRTL test_CapRTL_isolate.input  
+#+END_SRC
+* <2015-05-24 Sun>
+** Issues not covered in my current implementation
+   - In X2 and X3. Deal with overflow isolate count.
+   - Terminology
+     | Fribidi       | Unicode document         |
+     |---------------+--------------------------|
+     |               | overflow isolate count   |
+     | ~over_pushed~ | overflow embedding count |
+     | ~isolate~     | valid isolate count      |
+* <2015-05-29 Fri>
+** Comparing with the bidi ref test
+The following scenario was used for comparing with the bidi reference
+ - Download and compile the bidiref test with the following SConstruct file in file:../BidiReferenceC/6.3.0/source
+#+BEGIN_SRC python
+import glob
+
+env = Environment(CPPFLAGS=['-Wall','-g'],
+                  CPPPATH=['../include'])
+
+env.Program('bidiref',
+            glob.glob('*.c'))
+#+END_SRC 
+ - Run test suite on fribidi with:
+#+BEGIN_SRC sh
+cd ../fribidi-vs-unicode
+./test BidiTest.txt | head -20
+#+END_SRC 
+  - Create a file ~BidiTestSmall.txt~ that only contain the failing test. E.g.:
+#+BEGIN_EXAMPLE
+@Levels:	0 x x 1
+@Reorder:	0 3
+RLI LRE PDF R; 3
+#+END_EXAMPLE  
+  - Run the bidiref test on this single test with:
+#+BEGIN_SRC sh
+./bidiref -d4 -x BidiTestSmall.txt
+#+END_SRC 
+  - Note that bidiref contains magic to recognize the unicode file format only if the input file name contains the string "BidiTest".
+  - An example failing run for fribidi is:
+#+BEGIN_EXAMPLE
+> ./test b.txt 
+failure on line 4
+input is: RLI LRE PDF R; 3
+base dir: auto
+expected levels: 0 x x 1
+returned levels: 0 0 0 3
+expected order: 0 3
+returned order: 0 3
+#+END_EXAMPLE
+  - Goal is to debug, compare, and fix all ~BidiTestSmall.txt~ until the entire file passes.
+** Errors
+   - [X] Line 63038: RLI LRE PDF R; 3
+   - [ ] Line 66856: R RLI R  . 
+*** R RLI R
+    - Expected levels vs fribidi levels: 1 0 1 vs 1 1 1
+    - Don't understand the difference between the embedding level and the paragraph embedding level. 
+    - How can a level be lower than the paragraph level?
+* <2015-05-31 Sun>
+** Regarding R RLI R
+   - The problem is that N1 and N2 are done over isolating boundaries. This needs to be fixed! Either by remembering the isolating sequence level of each run and make sure to connect only runs of the same level, or alternatively to always sweep and count. Both of these solutions are slow as they require sweeping over inner runs. A faster solution would be to have a pointer to the next run of the same sequence. Can this be done efficiently?
+* <2015-06-05 Fri>
+  - [ ] make macros ~PREV_TYPE_SKIP_ISOLATE~ and ~NEXT_TYPE_SKIP_ISOLATE~ macros based on the code in N1.
+* <2015-06-18 Thu>
+** NSI merging problems
+   - The following ~b.txt~ shows that there is a difference between an isolated NSM and one preceded by an isolating sequence.
+#+BEGIN_EXAMPLE
+@Levels:	1
+@Reorder:	0
+NSM; 4
+
+@Levels:	1 1 1
+@Reorder:	2 1 0
+LRI PDI NSM; 4
+#+END_EXAMPLE   
+   - In both these cases, because of the RTL direction, the NSM should get the type of the base direction. This currently does not happen. See TBDov
+* <2015-06-20 Sat>
+** Test progress
+   | Date           | Num fail | First line fail | First fail                                 |
+   |----------------+----------+-----------------+--------------------------------------------|
+   | 2015-06-20 Sat |      139 |          236713 |                                            |
+   |                |       22 |          497052 | R ON FSI L PDI LRI L PDI RLI L PDI ON R; 2 |
+** Thoughts
+   - The quest of finding a single strategy for finding the next and previous types seem to fail. It seems like every rule is looking for something different. 
+   - The real problem is that sequences like ~R OL LRI ... PDI OL R~ should really be interpreted as ~R OL OL OL R~ for the algorithm to work. But in contrast to ~R OL OL OL~ that is compacted as ~R OL×3 R~ and which may be matched by a simple forward and backward match, in ~LRI ... PDI~ the right hand R must be found by scanning. This is what Ι tried doing in my changes to fribidi so far, but it failed due to differen needs.
+   - My strategy is first finding a scanning strategy that works, and only afterwards optimizing it.
+* <2015-06-21 Sun>
+** Test progress
+   | Date           | Num fail | First line fail | First fail                                      |
+   |----------------+----------+-----------------+-------------------------------------------------|
+   | 2015-06-21 Sat |       18 |          236713 | AL ON FSI L PDI LRI L PDI RLI R PDI ON ET EN; 2 |
+
+   - Note: I believe this is a bug...
+* <2015-08-23 Sun>
+** Misc
+   - All the unicode files should be moved to a common place, as ~BidiTest.txt~ and ~BidiCharacterTest.txt~ are needed for testing. This means moving out the ~UnicodeData.txt~ from ~gen.tab~ e.g. to ~.../fribid/Unidata~. 
+   - Create a new branch IsoLinks that adds FriBidiRun, ~next_iso~ and ~prev_iso~ for next and previous isolate runs.
+f   - ~test.c~ should probably be renamed to ~test-bidi-test-txt.c~ and ~test-character~ should be changed to ~test-bidi-character-test-txt.c~ to reflect what they test.
+* <2016-01-24 Sun>
+** Packing stuff
+   - [ ] Should we make sure that bootstrap and dependencies is clean? Currently lots of warnings on my system:
+#+BEGIN_SRC sh
+configure.ac:201: warning: LT_INIT was called before AM_PROG_AR
+/usr/share/aclocal-1.15/ar-lib.m4:13: AM_PROG_AR is expanded from...
+configure.ac:201: the top level
+configure.ac:201: warning: LT_INIT was called before AM_PROG_AR
+aclocal.m4:9317: AM_PROG_AR is expanded from...
+configure.ac:201: the top level
+libtoolize: putting auxiliary files in '.'.
+libtoolize: copying file './ltmain.sh'
+libtoolize: Consider adding 'AC_CONFIG_MACRO_DIRS([m4])' to configure.ac,
+libtoolize: and rerunning libtoolize and aclocal.
+libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am.
+configure.ac:201: warning: LT_INIT was called before AM_PROG_AR
+/usr/share/aclocal-1.15/ar-lib.m4:13: AM_PROG_AR is expanded from...
+configure.ac:201: the top level
+configure.ac:201: warning: LT_INIT was called before AM_PROG_AR
+aclocal.m4:9317: AM_PROG_AR is expanded from...
+configure.ac:201: the top level
+configure.ac:201: warning: LT_INIT was called before AM_PROG_AR
+aclocal.m4:9317: AM_PROG_AR is expanded from...
+configure.ac:201: the top level
+configure.ac:201: warning: LT_INIT was called before AM_PROG_AR
+aclocal.m4:9317: AM_PROG_AR is expanded from...
+configure.ac:201: the top level
+configure.ac:55: installing './compile'
+configure.ac:50: installing './missing'
+bin/Makefile.am: installing './depcomp'
+lib/Headers.mk:22: warning: shell cat $(top_srcdir: non-POSIX variable name
+lib/Headers.mk:22: (probably a GNU make extension)
+doc/Makefile.am:26:   'lib/Headers.mk' included from here
+lib/Headers.mk:22: warning: shell cat $(top_srcdir: non-POSIX variable name
+lib/Headers.mk:22: (probably a GNU make extension)
+lib/Makefile.am:28:   'lib/Headers.mk' included from here
+test/Makefile.am:30: warning: '%'-style pattern rules are a GNU make extension
+#+END_SRC   
+  - configure ok.
+  - make passes but outputs warnings about missing c2man.
+#+BEGIN_SRC sh
+WARNING: 'c2man' is missing on your system.
+         You might have modified some files without having the proper
+         tools for further handling them.  Check the 'README' file, it
+         often tells you about the needed prerequisites for installing
+         this package.  You may also peek at any GNU archive site, in
+         case some other package contains this missing 'c2man' program.
+#+END_SRC  
+  - Found c2man at http://www.ciselant.de/c2man/c2man.html , but it is not compatible with the make file usage.
+  - "make test" doesn't do anything.
+* <2016-01-25 Mon>
+** Release engineering
+   - Got "new" c2man from Ubuntu archive at https://answers.launchpad.net/ubuntu/+source/c2man/2.41-18
+   - Compiled as follows:
+#+BEGIN_SRC sh
+tar -xf ~/hd/Download/c2man_2.41.orig.tar.gz 
+mv c2man-2.41.orig c2man-2.41-18   
+cd c2man-2.41-18 
+zcat ~/hd/Download/c2man_2.41-18.diff.gz| patch -p1
+./Configure
+make install
+#+END_SRC   
+   - Verified that man pages are reasonoble (after fix to invocation of c2man).
+   - [ ] Wait for response from Behdad about the issues in mail I sent.
+   - [X] Rename unicode63 branch to dov-unicode63 branch.
+   - [X] Create new branch unicode63 out of master.
+   - [X] Manually cherry pick from dov-unicode63 changes.
+   - [X] Squash and rebase dov-unicode63 on top of unicode63.
+   - [ ] Pull request to Behdad (after got mail)
+** Testing
+#+BEGIN_SRC sh
+env LD_LIBRARY_PATH=../lib/.libs ./test BidiTest.txt 
+#+END_SRC
+  - Still 15 errors.
+* <2016-01-28 Thu>
+** Investigating failure
+   - Not clear to me the correctness of the following test:
+#+BEGIN_EXAMPLE
+@Levels:	0 0 x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x 124 124 124
+@Reorder:	0 1 66 67 68
+L WS LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO LRO RLO L L L; 2
+#+END_EXAMPLE
+  - Why doesn't the RLO raise the level to 125?
+  - Compare with Behdads file:../pybyedie . Oops! Doesn't support 6.3. :-(
+  - Found reason! There is one more LRO than Ι thought that wants to push the level to 126 which is illegal. We therefore get overflow which should prevent the RLO from increasing the level further!
+  - [ ] Figure out why this does not happen.
+* <2016-01-29 Fri>
+** Failure
+   - Added prevention of going from 124 → 125 in case of overflow. Solved 3 errors.
+   - [ ] Solve 12 remaining errors. :-)
+* <2016-01-30 Sat>
+** Compiling with debug
+#+BEGIN_SRC sh
+env CFLAGS='-fPIC -g -O0' ./configure --prefix=/usr/local --with-pic
+#+END_SRC  
+** Failures
+   - Next failure:
+#+BEGIN_EXAMPLE
+@Levels:	x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x 124 x 125 x 125 x x x x 125 x 125 x 124
+@Reorder:	62 73 71 66 64 75
+LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE ON RLO L LRE RLI LRE RLE LRO RLO PDI PDF L PDF ON; 1
+#+END_EXAMPLE
+   - The resulting discrepancy is:
+#+BEGIN_EXAMPLE
+        Exp Ret
+:
+61: LRE   x   0
+62: ON  124 124
+63: RLO   x 124
+64: L   125 125
+65: LRE   x 125
+66: RLI 125 125
+67: LRE   x 125
+68: RLE   x 125
+69: LRO   x 125
+70: RLO   x 125
+71: PDI 125   1
+72: PDF   x   1
+73: L   125 125
+74: PDF   x 125
+75: ON; 124 124
+#+END_EXAMPLE   
+   - Solved by 
+#+BEGIN_EXAMPLE
+@@ -539,7 +543,11 @@ fribidi_get_par_embedding_levels (
+           for (i = RL_LEN (pp); i; i--)
+             {
+               if (isolate_overflow > 0)
+-                isolate_overflow--;
++                {
++                  isolate_overflow--;
++                  RL_LEVEL (pp) = level;
++                }
+#+END_EXAMPLE      
+  - Next bug.
+#+BEGIN_EXAMPLE
+@Levels:	x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x 123 x 124 124 124 x x x x 124 124 124 x 123
+@Reorder:	75 64 65 66 71 72 73 62
+LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE RLE ON LRO R LRI R LRE RLE LRO RLO R PDI R PDF ON; 1
+#+END_EXAMPLE
+ - which yielded:
+#+BEGIN_EXAMPLE
+62: ON  123 123  
+63: LRO   x 123  
+64: R   124 124  
+65: LRI 124 124  
+66: R   124 124  
+67: LRE   x 124  
+68: RLE   x 124  
+69: LRO   x 124  
+70: RLO   x 124  
+71: R   124 125  (x)
+72: PDI 124 125  (x)
+73: R   124 125  (x)
+74: PDF   x 125  
+75: ON  123 124  (x)
+#+END_EXAMPLE
+  - The problem was that character 71 was increased even though we were in isolate mode. 
+  - Fixed!
+  - Next error:
+#+BEGIN_EXAMPLE
+failure on line 497576
+input is: LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE LRE RLE ON LRO R RLI ON LRO RLE RLO LRE ON PDI R PDF ON; 7
+base dir: auto
+expected levels: x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x 123 x 124 124 125 x x x x 125 124 124 x 123
+returned levels: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 123 123 124 124 125 125 125 125 125 125 125 125 125 125
+expected order: 75 64 65 71 66 72 73 62
+returned order: 64 65 75 73 72 71 66 62
+#+END_EXAMPLE    
+ - with output:
+#+BEGIN_EXAMPLE
+60: LRE   x   1  
+61: RLE   x   1  
+62: ON  123 123  
+63: LRO   x 123  
+64: R   124 124  
+65: RLI 124 124  
+66: ON  125 125  
+67: LRO   x 125        overflow++
+68: RLE   x 125        overflow++
+69: RLO   x 125        overflow++
+70: LRE   x 125        overflow++
+71: ON  125 125        
+72: PDI 124 125  (x)   should pop to RLI@65! -> 124
+73: R   124 125  (x)
+74: PDF   x 125        should pop to LRO@63 -> 123
+75: ON; 123 125  (x)
+#+END_EXAMPLE 
+  - The problem was that the PDI did not reset the overpushed level! Once it was inserted all tests pass.
+  - Patch that solved the problems:
+#+BEGIN_EXAMPLE
+@@ -281,6 +281,7 @@ print_bidi_string (
+ #define PUSH_STATUS \
+     FRIBIDI_BEGIN_STMT \
+       if LIKELY(over_pushed == 0 \
++                && isolate_overflow == 0 \
+                 && new_level <= FRIBIDI_BIDI_MAX_EXPLICIT_LEVEL)   \
+         { \
+           if UNLIKELY(level == FRIBIDI_BIDI_MAX_EXPLICIT_LEVEL - 1) \
+@@ -551,6 +552,7 @@ fribidi_get_par_embedding_levels (
+                      terminated by the PDI */
+                   while (stack_size && !status_stack[stack_size-1].isolate)
+                     POP_STATUS;
++                  over_pushed = 0; /* The PDI resets the overpushed! */
+                   POP_STATUS;
+                   isolate_level-- ;
+                   valid_isolate_count--;
+#+END_EXAMPLE    
diff --git a/test/Makefile.am b/test/Makefile.am
index ea7ac77..693d79d 100644
--- a/test/Makefile.am
+++ b/test/Makefile.am
@@ -27,7 +27,9 @@ EXTRA_DIST = \
 	$(TESTS) \
 	$(TEST_DATAS)
 
-test_%.reference: test_%.input
+SUFFIXES: .reference .input
+
+.reference.input:
 	testcase="$@" && \
 	charset="`echo "$@" | sed 's/_[^_]*$$//;s/.*_//'`" && \
 	$(top_builddir)/bin/fribidi --test --charset "$$charset" "$<" > "$@" \