summaryrefslogtreecommitdiff
path: root/enc
Commit message (Collapse)AuthorAgeFilesLines
* Update dependenciesNobuyoshi Nakada2019-11-181-43/+0
|
* Init function is need to link staticallyNobuyoshi Nakada2019-08-101-0/+5
|
* Removed unnecessary headersNobuyoshi Nakada2019-08-101-32/+0
|
* Use ENC_REPLICATE to copy an encodingNobuyoshi Nakada2019-08-101-8/+1
|
* Revert "Removed unused includes"Yusuke Endoh2019-08-102-0/+34
| | | | | | | | | | | | | | | | | | This reverts commit c9eb8f82e9febeb634a23bec6aeea915eb25fe26. The change caused "implicit declaration" warning and actual segfault. ``` /tmp/ruby/v2/src/trunk-gc-asserts/enc/gb2312.c: In function ‘Init_gb2312’: /tmp/ruby/v2/src/trunk-gc-asserts/enc/gb2312.c:6:31: warning: implicit declaration of function ‘rb_enc_find’ [-Wimplicit-function-declaration] rb_enc_register("GB2312", rb_enc_find("EUC-KR")); ^~~~~~~~~~~ /tmp/ruby/v2/src/trunk-gc-asserts/enc/gb2312.c:6:31: warning: passing argument 2 of ‘rb_enc_register’ makes pointer from integer without a cast [-Wint-conversion] <command-line>:0:19: note: expected ‘OnigEncoding {aka const struct OnigEncodingTypeST *}’ but argument is of type ‘int’ /tmp/ruby/v2/src/trunk-gc-asserts/regenc.h:231:12: note: in expansion of macro ‘ONIG_ENC_REGISTER’ extern int ONIG_ENC_REGISTER(const char *, OnigEncoding); ^~~~~~~~~~~~~~~~~ ```
* Removed unused includesNobuyoshi Nakada2019-08-092-34/+0
|
* Include ruby/assert.h in ruby/ruby.h so that assertions can be thereNobuyoshi Nakada2019-07-141-0/+24
|
* Update dependencies for 369ff79394765ce198ac7cee872a8c739d895aaaTakashi Kokubun2019-07-141-0/+10
| | | | | Just copy-pasting diff from https://travis-ci.org/ruby/ruby/jobs/558407687
* add encoding conversion from/to CESU-8Martin Dürst2019-07-141-0/+85
| | | | | | | | | | | | | | | Add encoding conversion (transcoding) from UTF-8 to CESU-8 and back. CESU-8 is an encoding similar to UTF-8, but encodes codepoints above U+FFFF as two surrogates, these surrogates again being encoded as if they were UTF-8 codepoints. This preserves the same binary sorting order as in UTF-16. It is also somewhat similar (although not exactly identical) to an encoding used internally by Java. This completes issue #15995. enc/trans/cesu_8.trans: Add encoding conversion from/to CESU-8 test/ruby/test_transcode.rb: Add tests for above
* Update dependenciesNobuyoshi Nakada2019-07-091-0/+7
|
* remove UNREACHABLENARUSE, Yui2019-06-241-2/+0
|
* Add new encoding CESU-8 [Feature #15931]NARUSE, Yui2019-06-241-0/+456
|
* remove Unicode 12.0.0 related directory and generated filesduerst2019-04-052-48546/+0
| | | | | | This completes issue #15195. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67453 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* update to Unicode Version 12.1.0 (beta)duerst2019-04-052-0/+49238
| | | | | | | | | | | | | | | | | | | | | | Unicode Version 12.1.0 adds one single character, U+32FF SQUARE ERA NAME REIWA, for the new Japanese era starting on May 1st. 12.1.0 will be finalized only on May 7th, so we go with the beta version because further changes in the data we need are highly unlikely, and we want to make sure Ruby is ready for the new era. * common.mk: change UNICODE_VERSION to 12.1.0, UNICODE_BETA to YES * enc/unicode/12.1.0, enc/unicode/12.1.0/casefold.h, enc/unicode/12.1.0/name2ctype.h: add directory and generated data files for new version * lib/unicode_normalize/tables.rb: update for new character * test/ruby/test_regexp.rb: add test for character property age=12.1 * test/test_unicode_normalize.rb: add test for NFKC decomposition of new character This (mostly) completes issue #15195. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67441 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* delete directory and files related to Unicode version 11.0.0duerst2019-03-062-47428/+0
| | | | | | this completes and closes feature #15321 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67174 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* update Unicode version (and Emoji version) to 12.0.0duerst2019-03-062-0/+48546
| | | | | | | | | | | | | - common.mk: set UNICODE_VERSION and UNICODE_EMOJI_VERSION to 12.0.0 - lib/unicode_normalize/tables.rb: update table data to Unicode version 12.0.0 - enc/unicode/12.0.0/casefold.h, enc/unicode/12.0.0/name2ctype.h: add generated files for Unicode version 12.0.0 This is the main commit for #15321. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67169 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Removed duplicate dependentsnobu2019-02-141-3/+0
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67072 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Update dependencies, internal.h includes ruby.hnobu2019-02-121-0/+1
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67057 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* implement special behavior for Georgian for String#capitalizeduerst2018-12-091-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The modern Georgian script is special in that it has an 'uppercase' variant called MTAVRULI which can be used for emphasis of whole words, for screamy headlines, and so on. However, in contrast to all other bicameral scripts, there is no usage of capitalizing the first letter in a word or a sentence. Words with mixed capitalization are not used at all. We therefore implement special behavior for String#capitalize. Formally, we define String#capitalize as first applying String#downcase for the whole string, then using titlecase on the first letter. Because Georgian defines titlecase as the identity function both for MTAVRULI ('uppercase') and Mkhedruli (lowercase), this results in String#capitalize being equivalent to String#downcase for Georgian. This avoids undesirable mixed case. * enc/unicode.c: Actual implementation * string.c: Add mention of this special case for documentation * test/ruby/enc/test_case_mapping.rb: Add two tests, a general one that uses String#capitalize on some (including nonsensical) combinations of MTAVRULI and Mkhedruli, and a canary test to detect the potential assignment of characters to the currently open slots (holes) at U+1CBB and U+1CBC. * test/ruby/enc/test_case_comprehensive.rb: Tweak generation of expectation data. Together with r65933, this closes issue #14839. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66300 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* delete Unicode 10.0.0 related files, no longer needed [#14802]duerst2018-12-092-45436/+0
| | | | | | | | | This line, and those below, will be ignored-- D enc/unicode/10.0.0 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66295 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* remove obsolete data from unicode.cduerst2018-12-061-26/+0
| | | | | | | | | | * unicode.c: Remove the arrays onigenc_unicode_GCB_ranges_GAZ, onigenc_unicode_GCB_ranges_E_Base, and onigenc_unicode_GCB_ranges_Emoji, because they are not needed anymore for Unicode 11.0.0. * regparse.c: Remove external declarations for above arrays. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66232 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* update to Unicode 11.0.0 (main step, not complete yet)duerst2018-12-052-0/+47428
| | | | | | | | | | | | | | - common.mk: Change Unicode version to 11.0.0, and Emoji version to 11.0 - test/ruby/enc/test_emoji_breaks.rb: update hard-coded Emoji version - enc/unicode/11.0.0, enc/unicode/11.0.0/casefold.h, enc/unicode/name2ctype.h: Add generated files. Files for Unicode 10.0.0 will be removed once we are sure 11.0.0 works. - lib/unicode_normalize/tables.rb: Updated table. - regparse.c: Almost completely reimplement grapheme cluster detection in function node_extended_grapheme_cluster(). git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66213 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* solve the genie/zombie/wrestlers bugduerst2018-12-021-8/+10
| | | | | | | | | | enc/unicode.c: - Add U+1F93C (WRESTLERS), U+1F9DE (GENIE), and U+1F9DF to onigenc_unicode_GCB_ranges_E_Base. - Add comments with character names. test/ruby/enc/test_emoji_breaks.rb: Activate tests for genie/zombie/wrestlers. This closes issue #15343. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66133 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Added words in the comment at r65088 [ci skip]nobu2018-11-301-2/+2
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66103 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Embed the Emoji versionnobu2018-11-271-0/+9
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66023 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* deal with ONIGENC_CASE_IS_TITLECASE flag on lowercase charactersduerst2018-11-251-4/+9
| | | | | | | | | In the function onigenc_unicode_case_map() in enc/unicode.c, deal with the case that the ONIGENC_CASE_IS_TITLECASE flag is set on lowercase characters. This is in preparation for Georgian Mtavruli, which are uppercase but not titlecase, in Unicode 11.0.0. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65971 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* prepare for Unicode 11.0.0 updateduerst2018-11-231-4/+3
| | | | | | | | | - enc/unicode/case-folding.rb: - Convert unpredicted case to actual flag setting - Eliminate an unused variable - Change a variable name to avoid a warning git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65933 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Make some internal functions staticnobu2018-11-161-2/+4
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65764 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* enc/unicode.c: 'a' is bigger than 'A'shyouhei2018-11-161-2/+4
| | | | | | | | | | | | | | | | | | | | | In ASCII, 'a' is bigger than 'A'. Which means 'A' - 'a' is a negative number (-32, to be precise). In C, the type of 'a' and 'A' are signed int (cf: ISO/IEC 9899:1990 section 6.1.3.4). So 'A' - 'a' is also a signed int. It is `(signed int)-32`. The problem is, OnigCodePoint is unsigned int. Adding a negative number to a variable of OnigCodepoint (`code` here) introduces an unintentional cast of `(unsigned)(signed)-32`, which is 4,294,967,264. Adding this value to code then overflows, and the result eventually becomes normal codepoint. The series of operations are not a serious problem but because `code >= 'a'` holds, we can `(code - 'a') + 'A'` to reroute this. See also: https://github.com/k-takata/Onigmo/pull/107 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65752 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* revert r65091, r65090 because ci failsduerst2018-10-164-47339/+14
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65093 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* update to Unicode 11.0.0 (basic step, not complete yet)duerst2018-10-164-5/+47330
| | | | | | | | | | | | | | | - common.mk: Change Unicode version to 11.0.0 - enc/unicode/case-folding.rb, enc/unicode.c: Initial changes to deal with Gregorian Mtavruli. This should bring us up to the same level as e.g. Python 3.7, by following the Unicode tables exactly. But it will produce undesirable (mixed-case) results for String#capitalize. This will be addressed in a later commit. - enc/unicode/11.0.0, enc/unicode/11.0.0/casefold.h, enc/unicode/name2ctype.h: Add generated files. - lib/unicode_normalize/tables.rb: Updated table. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65091 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* add some comments to enc/unicode/case-folding.rb [ci skip]duerst2018-10-161-9/+9
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65090 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Removed data for old Unicode [ci skip]nobu2018-10-161-28/+2
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65088 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* unicode.c: moved addtional GCB rangesnobu2018-10-151-0/+52
| | | | | | | * enc/unicode.c: moved additional Grapheme Cluster Break ranges which depend on the Unicode version. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65087 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* regparse.c: Suppress duplicated range warning by mere \Xnobu2018-10-151-2/+0
| | | | | | | | * regparse.c (node_extended_grapheme_cluster): as Unicode 10 has added Grapheme_Cluster_Break properties to some characters, remove duplicated ranges for Unicode 9. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65086 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby tool/update-deps --fixnaruse2018-07-051-0/+2
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63860 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Refactor ERB version checking for keyword argumentsk0kubun2018-02-271-2/+2
| | | | | | Improving code like r62590. See r62529 for details. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62594 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Judge ERB version not RUBY_VERSION but ERB.versionnaruse2018-02-241-2/+2
| | | | | | On cross compilation, ruby command uses fake RUBY_VERSION. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62560 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* erb.rb: deprecate safe_level of ERB.newk0kubun2018-02-221-2/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Also, as it's in the middle of the list of 4 arguments, 3rd and 4th arguments (trim_mode, eoutvar) are changed to keyword arguments. Old ways to specify arguments are deprecated and warned now. bin/erb: deprecate -S option. We'll remove all of deprecated ones at Ruby 2.7+. enc/make_encmake.rb: stopped using deprecated interface ext/etc/mkconstants.rb: ditto ext/socket/mkconstants.rb: ditto sample/ripper/ruby2html.rb: ditto spec/ruby/library/erb/defmethod/def_erb_method_spec.rb: ditto spec/ruby/library/erb/new_spec.rb: ditto test/erb/test_erb.rb: ditto test/erb/test_erb_command.rb: ditto tool/generic_erb.rb: ditto tool/ruby_vm/helpers/dumper.rb: ditto tool/transcode-tblgen.rb: ditto lib/rdoc/erbio.rb: ditto lib/rdoc/generator/darkfish.rb: ditto [Feature #14256] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62529 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Update dependenciesnobu2018-01-131-2/+0
| | | | | | | | | | | | | * common.mk: enc/unicode.$(OBJEXT) depends on onigmo.h via oniguruma.h. * common.mk: dependencies of *prelude.$(OBJEXT) are defined for each generated C sources. * enc/depend: casefold.h and name2ctype.h are located under $(UNICODE_HDR_DIR). git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61800 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Update dependencies using `tool/update-deps`kazu2018-01-131-0/+2
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61799 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* update dependenciesnobu2018-01-091-0/+2
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61715 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* gperf.sed: static declarationsnobu2017-12-153-3/+3
| | | | | | | * tool/gperf.sed: comment out arguments part only, to keep the following declarations static. [Feature #13883] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61282 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* support gperf 3.1nobu2017-12-084-42/+34
| | | | | | | | | | * tool/gperf.sed: extracted sed commands to a script. ANSI-C code produced by gperf 3.1 declares length arguments as `size_t`. it causes conflict with existing declarations, and needs casts for a local variable and return statements. [Feature #13883] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61076 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* fix for emoji-data.txtnobu2017-12-022-385/+653
| | | | | | | | | | | * common.mk: download emoji-data.txt. As emoji data files are located in a separate directory in Unicode.org site, reearranged Unicode data files directories same as the site. * tool/enc-unicode.rb (get_file): search emoji data files in the second argument path. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60977 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Add missing filenaruse2017-12-011-0/+546
| | | | git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60967 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* Update to Onigmo 6.1.3-669ac9997619954c298da971fcfacccf36909d05.naruse2017-12-012-523/+6
| | | | | | [Bug #13892] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60966 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* remove Unicode 9.0.0-related filesduerst2017-09-062-43881/+0
| | | | | | We don't need these files anymore because we upgraded to Unicode 10.0.0. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59760 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* update Ruby to Unicode 10.0.0duerst2017-09-062-0/+45155
| | | | | | | | | - In common.mk, set UNICODE_VERSION to 10.0.0 - Generate and add enc/unicode/10.0.0/casefold.h and enc/unicode/10.0.0/name2ctype.h - Update lib/unicode_normalize/tables.rb git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59759 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* replace copyrights by explanatory text in data files for GB2312/GB12345 mappingsduerst2017-09-014-252/+20
| | | | | | | Replace the copyrights and explanatory texts in the data files used for mapping GB2312/GB12345 to/from Unicode with short explanatory texts. [Bug #12598] [ci skip] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59715 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
View Lorry Controller Status