summaryrefslogtreecommitdiff
path: root/sv.c
Commit message (Collapse)AuthorAgeFilesLines
* Move case change invlists from interpreter to globalKarl Williamson2018-03-261-5/+0
| | | | | These are now constant through the life of the program, so don't need to be duplicated at each new thread instantiation.
* Move UTF-8 case changing data into coreKarl Williamson2018-03-261-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Prior to this commit, if a program wanted to compute the case-change of a character above 0xFF, the C code would switch to perl, loading lib/utf8heavy.pl and then read another file from disk, and then create a hash. Future references would use the hash, but the start up cost is quite large. There are five case change types, uc, lc, tc, fc, and simple fc. Only the first encountered requires loading of utf8_heavy, but each required switching to utf8_heavy, and reading the appropriate file from disk. This commit changes these functions to use compiled-in C data structures (inversion maps) to represent the data. To look something up requires a binary search instead of a hash lookup. An individual hash lookup tends to be faster than a binary search, but the differences are small for small sizes. I did some benchmarking some years ago, (commit message 87367d5f9dc9bbf7db1a6cf87820cea76571bf1a) and the results were that for fewer than 512 entries, the binary search was just as fast as a hash, if not actually faster. Now, I've done some more benchmarks on blead, using the tool benchmark.pl, which wasn't available back then. The results below indicate that the differences are minimal up through 2047 entries, which all Unicode properties are well within. A hash, PL_foldclosures, is still constructed at runtime for the case of regular expression /i matching, and this could be generated at Perl compile time, as a further enhancement for later. But reading a file from disk is no longer required to do this. ======================= benchmarking results ======================= Key: Ir Instruction read Dr Data read Dw Data write COND conditional branches IND indirect branches _m branch predict miss _m1 level 1 cache miss _mm last cache (e.g. L3) miss - indeterminate percentage (e.g. 1/0) The numbers represent raw counts per loop iteration. "\x{10000}" =~ qr/\p{CWKCF}/" swash invlist Ratio % fetch search ------ ------- ------- Ir 2259.0 2264.0 99.8 Dr 665.0 664.0 100.2 Dw 406.0 404.0 100.5 COND 406.0 405.0 100.2 IND 17.0 15.0 113.3 COND_m 8.0 8.0 100.0 IND_m 4.0 4.0 100.0 Ir_m1 8.9 17.0 52.4 Dr_m1 4.5 3.4 132.4 Dw_m1 1.9 1.2 158.3 Ir_mm 0.0 0.0 100.0 Dr_mm 0.0 0.0 100.0 Dw_mm 0.0 0.0 100.0 These were constructed by using the file whose contents are below, which uses the property in Unicode that currently has the largest number of entries in its inversion list, > 1600. The test was run on blead -O2, no debugging, no threads. Then the cut-off boundary was changed from 512 to 2047 for when we use a hash vs an inversion list, and the test run again. This yields the difference between a hash fetch and an inversion list binary search ===================== The benchmark file is below =============== no warnings 'once'; my @benchmarks; push @benchmarks, 'swash' => { desc => '"\x{10000}" =~ qr/\p{CWKCF}/"', setup => 'no warnings "once"; my $re = qr/\p{CWKCF}/; my $a = "\x{10000}";', code => '$a =~ $re;', }; \@benchmarks;
* Don't include interpreter variable unless usedKarl Williamson2018-03-161-0/+2
| | | | | | | This adds an #ifdef around this variable, so that it isn't defined unless used. Spotted by Daniel Dragan.
* Make Unicode data structures globalKarl Williamson2018-03-141-20/+0
| | | | | | | | | | These structures are read-only, use const C strings, and are truly global, so no need to have them be interpreter level. This saves duplicating and freeing them as threads come and go. In doing this, I noticed that not every one was properly being copied/deallocated, so this fixes some potential unreported bugs, and leaks.
* Silence wrong clang warningsKarl Williamson2018-03-011-0/+5
| | | | | Clang thread-safety analysis fails to correctly work in this situation (and is documented as failing), so turn off that warning here.
* remove unused var in sv_eq_flagsDaniel Dragan2018-03-011-7/+3
| | | | | | | | | | | | | svrecode became unused in commit 8df0e7a28b "Remove IN_ENCODING macro, and all code dependent on it" but there was still a SvREFCNT_dec(NULL) executed at the end of the function. This commit will reduce size of Perl_sv_eq_flags by the CC not having to save var eq to a non-volatile register or stack location around the SvREFCNT_dec func call and instead store var eq in the return register directly. Also remove the eq var completly, since initializing the var so early means it has to be stored on the stack around alot func calls, so just do a direct return of const zero on the only "fall off the end" path in the func.
* Add thread-safe locale handlingKarl Williamson2018-02-181-0/+7
| | | | | | This (large) commit allows locales to be used in threaded perls on platforms that support it. This includes recent Windows and Posix 2008 ones.
* Latch LC_NUMERIC during critical sectionsKarl Williamson2018-02-181-0/+1
| | | | | | | | | | | | | | It is possible for operations on threaded perls which don't 'use locale' to still change the locale. This happens when calling POSIX::localeconv() and I18N::Langinfo(), and in earlier perls, it can happen for other operations when perl has been initialized with the environment causing the various locale categories to not have a uniform locale. This commit causes the areas where the locale for this category should predictably be in one or the other state to be a critical section where another thread can't interrupt and change it. This is a separate mutex, so that only these particular operations will be held up.
* Add Perl_setlocale()Karl Williamson2018-02-181-0/+3
| | | | | | | | | | | | | | | | | | khw could not find any modules on CPAN that correctly use the C library function setlocale(). (The very few that do try, do not use it correctly, looking at the return value incorrectly, so they are broken.) This analysis does not include modules that call non-Perl libaries that may call setlocale(). And, a future commit will render the setlocale() function useless in some configurations on some platforms. So this commit adds Perl_setlocale(), for XS code to call, and which is always effective, but it should not be used to alter the locale except on platforms where the predefined variable ${^SAFE_LOCALES} evaluates to 1. This function is also what POSIX::setlocale() calls to do the real work.
* Use proper #define to see if need PLnumeric underlying_objKarl Williamson2018-02-181-1/+1
| | | | | perl.h has a single #define which is the combination of several that determines if this object should be created or not.
* Avoid changing locale when finding radix charKarl Williamson2018-01-301-0/+4
| | | | | | | | | On systems that have the POSIX 2008 operations, including nl_langinfo_l(), this commit causes them to not have to actually change the locale when determining what the decimal point character is. The locale may have to change during the printing/reading of numbers, but eventually we can use sprintf_l(), if available, to avoid that too.
* Perl_sv_2pv_flags: Potentially avoid workKarl Williamson2018-01-301-1/+1
| | | | | | By using a macro that is private to the core, this code can avoid thinking it has to deal with a non-dot radix character, as even if we are using the locale radix, that is often a dot.
* Keep PL_numeric_radix_sv always setKarl Williamson2018-01-301-2/+2
| | | | | Previously this was removed if the radix was dot. By keeping it set to a dot, we simplify some code, removing some branches.
* sv_vcatpvfn_flags() Balance LC_NUMERIC changes/restoresKarl Williamson2018-01-301-2/+4
| | | | | | Prior to this commit, the restore for LC_NUMERIC was getting called even if there were no corresponding store. Change so they are balanced; a future commit will require this.
* Simplify some LC_NUMERIC macrosKarl Williamson2018-01-301-4/+6
| | | | | | | These macros are marked as subject to change and are not documented externally. I don't know what I was thinking when I named some of them, but whatever no longer makes sense to me. Simplify them, and change so there is only one restore macro to remember.
* Cache locale UTF8-ness lookupsKarl Williamson2018-01-301-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some locales are UTF-8, some are not. Knowledge of this is needed in various circumstances. This commit saves the results of the last several lookups so they don't have to be recalculated each time. The full generality of POSIX locales is such that you can have error messages be displayed in one locale, say Spanish, while other things are in French. To accommodate this generality, the program can loop through all the locale categories finding the UTF8ness of the locale it points to. However, in almost all instances, people are going to be in either French or in Spanish, and not in some combination. Suppose it is a French UTF-8 locale for all categories. This new cache will know that the French locale is UTF-8, and the queries for all but the first category can return that immediately. This simple cache avoids the overhead of hashes. This also fixes a bug I realized exists in threaded perls, but haven't reproduced. We do not support locales in such perls, and the user must not change the locale or 'use locale'. But perl itself could change the locale behind the scenes, leading to segfaults or incorrect results. One such instance is the determination of UTF8ness. But this only could happen if the full generality of locales is used so that the categories are not all in the same locale. This could only happen (if the user doesn't change locales) if the environment is such that the perl program is started up so that the categories are in such a state. This commit fixes this potential bug by caching the UTF8ness of each category at startup, before any threads are instantiated, and so checking for it later just looks it up in the cache, without perl changing the locale.
* Avoid some unnecessary changing of localesKarl Williamson2018-01-301-0/+1
| | | | | | | | | | The LC_NUMERIC locale category is kept so that generally the decimal point (radix) is a dot. For some (mostly) output purposes, it needs to be swapped into the program's current underlying locale so that a non-dot can be printed. This commit changes things so that if the current underlying locale uses a decimal point, the swap doesn't happen, as it's not needed.
* newSVpvn(): Fix podKarl Williamson2018-01-191-1/+1
| | | | | | There is no "buffer" argument; don't refer to one. Spotted by KES
* fix F0convert() on edge casesZefram2018-01-191-4/+7
| | | | | | | | | | The F0convert() function used to implement the %.0f format specifier more cheaply went wrong on some edge cases. Its rounding went wrong when the exponent is such that fractional values are not representable, making the "+= 0.5" invoke floating point rounding. Fix that by only invoking that rounding logic for values that start out fractional. That fixes the output part of [perl #47602]. It also failed to emit the sign for negative zero. Fix that by making it not apply to zero values.
* Fix stray > in L<perlapi/sv_usepvn_flags>Dagfinn Ilmari Mannsåker2018-01-161-1/+1
|
* revert smartmatch to 5.27.6 behaviourZefram2017-12-291-2/+5
| | | | | | | | | | | | | The pumpking has determined that the CPAN breakage caused by changing smartmatch [perl #132594] is too great for the smartmatch changes to stay in for 5.28. This reverts most of the merge in commit da4e040f42421764ef069371d77c008e6b801f45. All core behaviour and documentation is reverted. The removal of use of smartmatch from a couple of tests (that aren't testing smartmatch) remains. Customisation of a couple of CPAN modules to make them portable across smartmatch types remains. A small bugfix in scope.c also remains.
* Add script_run regex featureKarl Williamson2017-12-241-0/+1
| | | | As explained in the docs, this helps detect spoofing attacks.
* narrow scope of argsv in sv_vcatpvfn_flags()Zefram2017-12-231-1/+1
| | | | | | argsv was being retained between format specifiers, causing incorrect treatment of format specifiers following one that took an SV argument. Fixes [perl #132645].
* merge branch zefram/dumb_matchZefram2017-12-171-5/+2
|\
| * internally change "when" to "whereso"Zefram2017-12-051-1/+1
| | | | | | | | | | The names of ops, context types, functions, etc., all change in accordance with the change of keyword.
| * make loop control apply to "given"Zefram2017-11-291-4/+1
| | | | | | | | A "given" construct is now officially a one-iteration loop.
| * use blk_loop format for CXt_GIVENZefram2017-11-291-2/+2
| |
* | semicolon-friendly diagnostic controlZefram2017-12-161-4/+4
| | | | | | | | | | | | New macros {GCC,CLANG}_DIAG_{IGNORE,RESTORE}_{DECL,STMT}, which take a following semicolon. It is necessary to use the _DECL or _STMT version as appropriate to the context. Fixes [perl #130726].
* | make exec keep its argument list more reliablyZefram2017-12-141-2/+0
| | | | | | | | | | | | | | | | | | | | Bits of exec code were putting the constructed commands into globals PL_Argv and PL_Cmd, which could then be clobbered by reentrancy. These are only global in order to manage their freeing, but that's better managed by using the scope stack. So replace them with automatic variables, with ENTER/SAVEFREEPV/LEAVE to free the memory. Also copy the strings acquired from SVs, to avoid magic clobbering the buffers of SVs already read. Fixes [perl #129888].
* | sv.c: White-space onlyKarl Williamson2017-12-111-10/+10
| | | | | | | | | | This outdents some lines that the previous commit removed an enclosing block of.
* | utf8_upgrade_flags_grow(): Use faster variant countKarl Williamson2017-12-111-168/+64
| | | | | | | | | | | | | | | | | | Now that we have a much faster way of counting the characters that would expand to two bytes when a string is translated into UTF-8, use that, instead of trying to choose one of two methods. This simplifies the code. Note that the faster method doesn't happen on EBCDIC platforms, but the simplification is worth the potential loss of being able to choose methods.
* | Allow %j on non-C99 platformsKarl Williamson2017-12-081-11/+3
| | | | | | | | | | Now that intmax_t is emulated, the %j format is usable on platforms that aren't C99.
* | assert() that the vlnz is not NULL before using it.Jarkko Hietaniemi2017-11-291-0/+1
| | | | | | | | Coverity #169272.
* | sv_utf8_decode: Reverse order of tests for speedKarl Williamson2017-11-241-4/+4
|/ | | | | | | | | Not that we have a fast is_utf8_invariant_string_loc(), use it first to quickly find any variants. Then use is_utf8_string() from then on. This is the reverse order as to how it worked before this commit. This speeds things up two ways: 1) we use the faster function first, and 2) use the information it returns to avoid reparsing the string starting at the beginning.
* avoid redundant initialisation around Newxz()Zefram2017-11-131-14/+9
| | | | | | Reduce Newxz() to Newx() where all relevant parts of the memory are being explicitly initialised, and don't explicitly zero memory that was already zeroed. [perl #36078]
* duplicate full index for SAVEt_AELEMZefram2017-11-131-2/+2
| | | | | | | | The index in a SAVEt_AELEM save entry is now IV-sized, but only an I32 portion of it was being duplicated. This would lead to restoring the wrong element if a pseudfork were done with a localised array element on the stack, if the array index were above the 32-bit range or on a big-endian architecture.
* sv.c: Fix typo in commentKarl Williamson2017-11-091-1/+1
|
* Change name of locale per-interpreter variableKarl Williamson2017-11-081-2/+2
| | | | | | The real purpose of this internal variable is to give the name of the locale that is the underlying one for the C program. Various macros already indicate that. This furthers the process.
* Use memEQs, memNEs in core filesKarl Williamson2017-11-061-4/+3
| | | | | | | | | | Where the length is known, we can use these functions which relieve the programmer and the program reader from having to count characters. The memFOO functions should also be slightly faster than the strFOO equivalents. In some instances in this commit, hard coded numbers are used. These come from the 'case' statement values that apply to them.
* Add OP_MULTICONCAT opDavid Mitchell2017-10-311-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Allow multiple OP_CONCAT, OP_CONST ops, plus optionally an OP_SASSIGN or OP_STRINGIFY, to be combined into a single OP_MULTICONCAT op, which can make things a *lot* faster: 4x or more. In more detail: it will optimise into a single OP_MULTICONCAT, most expressions of the form LHS RHS where LHS is one of (empty) my $lexical = $lexical = $lexical .= expression = expression .= and RHS is one of (A . B . C . ...) where A,B,C etc are expressions and/or string constants "aAbBc..." where a,A,b,B etc are expressions and/or string constants sprintf "..%s..%s..", A,B,.. where the format is a constant string containing only '%s' and '%%' elements, and A,B, etc are scalar expressions (so only a fixed, compile-time-known number of args: no arrays or list context function calls etc) It doesn't optimise other forms, such as ($a . $b) . ($c. $d) ((($a .= $b) .= $c) .= $d); (although sub-parts of those expressions might be converted to an OP_MULTICONCAT). This is partly because it would be hard to maintain the correct ordering of tie or overload calls. The compiler uses heuristics to determine when to convert: in general, expressions involving a single OP_CONCAT aren't converted, unless some other saving can be made, for example if an OP_CONST can be eliminated, or in the presence of 'my $x = .. ' which OP_MULTICONCAT can apply OPpTARGET_MY to, but OP_CONST can't. The multiconcat op is of type UNOP_AUX, with the op_aux structure directly holding a pointer to a single constant char* string plus a list of segment lengths. So for "a=$a b=$b\n"; the constant string is "a= b=\n", and the segment lengths are (2,3,1). If the constant string has different non-utf8 and utf8 representations (such as "\x80") then both variants are pre-computed and stored in the aux struct, along with two sets of segment lengths. For all the above LHS types, any SASSIGN op is optimised away. For a LHS of '$lex=', '$lex.=' or 'my $lex=', the PADSV is optimised away too. For example where $a and $b are lexical vars, this statement: my $c = "a=$a, b=$b\n"; formerly compiled to const[PV "a="] s padsv[$a:1,3] s concat[t4] sK/2 const[PV ", b="] s concat[t5] sKS/2 padsv[$b:1,3] s concat[t6] sKS/2 const[PV "\n"] s concat[t7] sKS/2 padsv[$c:2,3] sRM*/LVINTRO sassign vKS/2 and now compiles to: padsv[$a:1,3] s padsv[$b:1,3] s multiconcat("a=, b=\n",2,4,1)[$c:2,3] vK/LVINTRO,TARGMY,STRINGIFY In terms of how much faster it is, this code: my $a = "the quick brown fox jumps over the lazy dog"; my $b = "to be, or not to be; sorry, what was the question again?"; for my $i (1..10_000_000) { my $c = "a=$a, b=$b\n"; } runs 2.7 times faster, and if you throw utf8 mixtures in it gets even better. This loop runs 4 times faster: my $s; my $a = "ab\x{100}cde"; my $b = "fghij"; my $c = "\x{101}klmn"; for my $i (1..10_000_000) { $s = "\x{100}wxyz"; $s .= "foo=$a bar=$b baz=$c"; } The main ways in which OP_MULTICONCAT gains its speed are: * any OP_CONSTs are eliminated, and the constant bits (already in the right encoding) are copied directly from the constant string attached to the op's aux structure. * It optimises away any SASSIGN op, and possibly a PADSV op on the LHS, in all cases; OP_CONCAT only did this in very limited circumstances. * Because it has a holistic view of the entire concatenation expression, it can do the whole thing in one efficient go, rather than creating and copying intermediate results. pp_multiconcat() goes to considerable efforts to avoid inefficiencies. For example it will only SvGROW() the target once, and to the exact size needed, no matter what mix of utf8 and non-utf8 appear on the LHS and RHS. It never allocates any temporary SVs except possibly in the case of tie or overloading. * It does all its own appending and utf8 handling rather than calling out to functions like sv_catsv(). * It's very good at handling the LHS appearing on the RHS; for example in $x = "abcd"; $x = "-$x-$x-"; It will do roughly the equivalent of the following (where targ is $x); SvPV_force(targ); SvGROW(targ, 11); p = SvPVX(targ); Move(p, p+1, 4, char); Copy("-", p, 1, char); Copy("-", p+5, 1, char); Copy(p+1, p+6, 4, char); Copy("-", p+10, 1, char); SvCUR(targ) = 11; p[11] = '\0'; Formerly, pp_concat would have used multiple PADTMPs or temporary SVs to handle situations like that. The code is quite big; both S_maybe_multiconcat() and pp_multiconcat() (the main compile-time and runtime parts of the implementation) are over 700 lines each. It turns out that when you combine multiple ops, the number of edge cases grows exponentially ;-)
* get rid of "implicit fallthrough" warnings with gcc 7Lukas Mai2017-10-221-0/+1
|
* Fix sv_vcatpvfn %s with precision on non-NUL-terminated stringsDagfinn Ilmari Mannsåker2017-10-211-1/+4
| | | | | | The precision parameter to %s can be used to print non-NUL-terminated strings, so use my_strnlen() to limit the length checking to the specified length.
* sv.c: simplify cpp conditionalsAaron Crane2017-10-211-3/+1
|
* Use snprintf() in favour of sprintf()Aaron Crane2017-10-211-1/+1
|
* Use preprocessor check for some DEBUG_X_TESTNicolas R2017-10-161-7/+15
| | | | | | | | | Most of the DEBUG_?_TEST calls are already protected by one '#idef DEBUGGING' check, but noticed a few of them which were not protected in sv.c and toke.c We should avoid these extra 'if' statements if perl is not compiled with debug option: -DDEBUGGING.
* avoid gcc 7 warning about * in boolean contextLukas Mai2017-10-131-2/+2
|
* get rid of "implicit fallthrough" warnings with gcc 7Lukas Mai2017-10-131-0/+3
|
* sv.c: Use modern macro namesKarl Williamson2017-09-091-4/+4
| | | | | This used older names for some locale functions, that have been superseded by more meaningful ones.
* Add API function Perl_langinfo()Karl Williamson2017-09-091-0/+3
| | | | | | This is designed to generally replace nl_langinfo() in XS code. It is thread-safer, hides the quirks of perl's LC_NUMERIC handling, and can be used on systems lacking nl_langinfo.
* Add new API function sv_rvunweakenDagfinn Ilmari Mannsåker2017-09-041-1/+38
| | | | | | | Needed to fix in-place sort of weak references in a future commit. Stolen from Scalar::Util::unweaken, which will be made to use this when available via CPAN upstream.
View Lorry Controller Status