+ support UTF-8 in ascii2unicode(), this fixes the UTF-16 output of

    resourcestring data .rsj files in case the source file is interpreted as
    UTF-8. Previously, the individual UTF-8 bytes were each stored in a
    separate widechar in the Json file (mantis #28717)

  * due to the fact that rstconv didn't use the cwstring unit on Unix, rstconv
    until now just concatenated the bytes stored in the widechars of the Json
    file on those platforms, i.e., the strings put in the resource file were
    byte for byte equal to what was in the source file. On Windows, these bytes
    were interpreted as individual widechars, converted to the
    DefaultSystemCodePage and then written. This means that for anything but
    ISO-8859-1 (where every widechar from #0000 to #0255 maps to #0 to #255),
    the output got corrupted.

    In order to keep compatibility with the old behaviour whereby rstconv wrote
    the resource strings using the same encoding as in the source file (except
    if the data got completely corrupted, in which case compatibility is
    useless), we now store all resourcestrings twice in the .rsj file: once as
    the exact byte sequence from the source file, and once (properly) encoded
    in UTF-16.

    By default, rstconv will use the byte string and just write that one to the
    resource file. Additionally, there is a new -p option that accepts a code
    page name (see rstconv -h for the list of supported names), which can be
    used to make rstconv use the UTF-16 version and convert that to the desired
    code page (as long as the system on which rstconv runs supports that
    codepage).

    And this also finally resolves mantis #6477.

git-svn-id: http://svn.freepascal.org/svn/fpc/trunk@31881 3ad0048d-3df7-0310-abae-a5850022a9f2

1 files changed, 12 insertions, 1 deletions

diff --git a/compiler/cresstr.pas b/compiler/cresstr.pas
index a481485de7..456c743f45 100644
--- a/compiler/cresstr.pas
+++ b/compiler/cresstr.pas
@@ -237,11 +237,22 @@ uses
             message1(general_e_errorwritingresourcefile,ResFileName);
             exit;
           end;
+        { write the data in two formats:
+           a) backward compatible: the plain bytes from the source file
+           b) portable: converted to utf-16
+        }
         writeln(f,'{"version":1,"strings":[');
         R:=TResourceStringItem(List.First);
         while assigned(R) do
           begin
-            write(f, '{"hash":',R.Hash,',"name":"',R.Name,'","value":"');
+            write(f, '{"hash":',R.Hash,',"name":"',R.Name,'","sourcebytes":[');
+            for i:=0 to R.Len-1 do
+              begin
+                write(f,ord(R.Value[i]));
+                if i<>R.Len-1 then
+                  write(f,',');
+              end;
+            write(f,'],"value":"');
             initwidestring(W);
             ascii2unicode(R.Value,R.Len,current_settings.sourcecodepage,W);
             for I := 0 to W^.len - 1 do