diff options
author | Kirill Smelkov <kirr@mns.spb.ru> | 2013-03-07 14:55:07 +0400 |
---|---|---|
committer | Junio C Hamano <gitster@pobox.com> | 2013-03-09 11:11:19 -0800 |
commit | 6cd3c0532772749bcf6c4688f34c8ecc65ecb655 (patch) | |
tree | e9cbd620a7e93668a04d87b2c0fd1a2017e2b3d5 /utf8.c | |
parent | 15999998fbda60552742275570947431b57108ae (diff) | |
download | git-6cd3c0532772749bcf6c4688f34c8ecc65ecb655.tar.gz |
format-patch: RFC 2047 says multi-octet character may not be split
Even though an earlier attempt (bafc478..41dd00bad) cleaned
up RFC 2047 encoding, pretty.c::add_rfc2047() still decides
where to split the output line by going through the input
one byte at a time, and potentially splits a character in
the middle. A subject line may end up showing like this:
".... fö?? bar". (instead of ".... föö bar".)
if split incorrectly.
RFC 2047, section 5 (3) explicitly forbids such beaviour
Each 'encoded-word' MUST represent an integral number of
characters. A multi-octet character may not be split across
adjacent 'encoded- word's.
that means that e.g. for
Subject: .... föö bar
encoding
Subject: =?UTF-8?q?....=20f=C3=B6=C3=B6?=
=?UTF-8?q?=20bar?=
is correct, and
Subject: =?UTF-8?q?....=20f=C3=B6=C3?= <-- NOTE ö is broken here
=?UTF-8?q?=B6=20bar?=
is not, because "ö" character UTF-8 encoding C3 B6 is split here across
adjacent encoded words.
To fix the problem, make the loop grab one _character_ at a time and
determine its output length to see where to break the output line. Note
that this version only knows about UTF-8, but the logic to grab one
character is abstracted out in mbs_chrlen() function to make it possible
to extend it to other encodings with the help of iconv in the future.
Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Diffstat (limited to 'utf8.c')
-rw-r--r-- | utf8.c | 39 |
1 files changed, 39 insertions, 0 deletions
@@ -495,3 +495,42 @@ char *reencode_string(const char *in, const char *out_encoding, const char *in_e return out; } #endif + +/* + * Returns first character length in bytes for multi-byte `text` according to + * `encoding`. + * + * - The `text` pointer is updated to point at the next character. + * - When `remainder_p` is not NULL, on entry `*remainder_p` is how much bytes + * we can consume from text, and on exit `*remainder_p` is reduced by returned + * character length. Otherwise `text` is treated as limited by NUL. + */ +int mbs_chrlen(const char **text, size_t *remainder_p, const char *encoding) +{ + int chrlen; + const char *p = *text; + size_t r = (remainder_p ? *remainder_p : SIZE_MAX); + + if (r < 1) + return 0; + + if (is_encoding_utf8(encoding)) { + pick_one_utf8_char(&p, &r); + + chrlen = p ? (p - *text) + : 1 /* not valid UTF-8 -> raw byte sequence */; + } + else { + /* + * TODO use iconv to decode one char and obtain its chrlen + * for now, let's treat encodings != UTF-8 as one-byte + */ + chrlen = 1; + } + + *text += chrlen; + if (remainder_p) + *remainder_p -= chrlen; + + return chrlen; +} |