| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
| |
This is now used as a cache of length 1 to avoid having to lookup up the
UTF-8ness as often.
This commit also skips doing S_newctype() if the new boss is the same as
the old
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This env var can be used to trigger a repeatable run of a script which
calls C<srand()> with no arguments, either explicitly or implicitly
via use of C<rand()> prior to calling srand(). This is implemented in
such a way that calling C<srand()> with no arguments in forks or
subthreads (again explicitly or implicitly) will receive their own seed
but the seeds they receive will be repeatable.
This is intended for debugging and perl development performance testing,
and for running the test suite consistently. It is documented that the
exact seeds used to initialize the random state are unspecified, and
that they may change between releases or even builds. The only guarantee
provided is that the same perl executable will produce the same results
twice all other things being equal. In practice and in core testing we
do expect consistency, but adding the tightest set of restrictions on
our commitments seemed sensible.
The env var is ignored when perl is run setuid or setgid similarly to
the C<PERL_INTERNAL_RAND_SEED> env var.
|
|
|
|
|
|
|
|
|
|
|
|
| |
The POSIX 2008 API has an edge case in that the result of most of the
functions when called with a global (as opposed to a per-thread) locale
is undefined.
The duplocale() function is the exception which will create a per-thread
locale containing the values copied from the global one.
This commit just calls duplocale, if needed, and the caller need not
concern itself with this possibility
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This function is rewritten to handle LC_ALL, and to make it easier to
add new checks.
There is also a change, which I think is an improvement, in that everything
starting with a \n is trimmed, instead of just a trailing \n.
A couple of calls to stdize_locale() are removed, as they are redundant,
because they are called only as a result of Perl_setlocale() being
called, and that ends up calling stdize_locale always, early on.
The call to savepv() is also moved in a couple cases to after the result
is known to not be NULL
I originally had such a new check in mind, but it turned out that doing
it here didn't solve the problem, so this commit has been amended
(before ever being pushed) to not include that.
chomped.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A long standing bug in Perl that has gone undetected is that the array
is global that is created when changing locales and tells fc() and qr//i
matching what the folds are in the new locale.
What this means is that any program only has one set of fold definitions
that apply to all threads within it, even if we claim that the locales
are thread-safe on the given platform. One possibility for this going
undetected so long is that no one is using locales on multi-threaded
systems much. Another possibility is that modern UTF-8 locales have the
same set of folds as any other one.
It is a simple matter to make the fold array per-thread instead of
per-process, and that solves the problem transparently to other code.
I discovered this stress-testing locale handling under threads. That
test will be added in a future commit.
In order to keep from having a dTHX inside foldEQ_locale, it has to have
a pTHX_ parameter. This means that the other functions that function
pointer variables get assigned to point to have to have an identical
signature, which means adding pTHX_ to functions that don't require it.
The bodies of all these are known to the compiler, since they are all
in inline.h or in the same .c file as where they are called. Hence the
compiler can optimize out the unused parameter.
Two calls of STR_WITH_LEN also have to be changed because of C
preprocessor limitations; perhaps there is another way to do it that I'm
unfamiliar with.
|
|
|
|
|
| |
The purpose of PL_origenviron is to preserve the earliest known value
of environ, which is a global. All interpreters should share it.
|
|
|
|
|
|
| |
Save/restore PL_prevailing_version at SAVEHINTS time
Have PL_prevailing_version track the applied use VERSION currently in scope
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since the removal of PERL_OBJECT
(acfe0abcedaf592fb4b9cb69ce3468308ae99d91) PERL_IMPLICIT_CONTEXT and
MULTIPLICITY have been synonymous and they're being used interchangeably.
To simplify the code, this commit replaces all instances of
PERL_IMPLICIT_CONTEXT with MULTIPLICITY.
PERL_IMPLICIT_CONTEXT will stay defined for compatibility with XS
modules.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This fixes GH #18341
There are problems with getenv() on threaded perls wchich can lead to
incorrect results when compiled with PERL_MEM_LOG.
Commit 0b83dfe6dd9b0bda197566adec923f16b9a693cd fixed this for some
platforms, but as Tony Cook, pointed out there may be
standards-compliant platforms that that didn't fix.
The detailed comments outline the issues and (complicated) full solution.
|
|
|
| |
Co-authored-by: Karl Williamson <khw@cpan.org>
|
|
|
|
|
|
|
|
| |
This was originally added for MinGW, which no longer needs it, and
only still used by Symbian, which is now removed.
This also leaves perlapi.[ch] empty, but we keep the header for CPAN
backwards compatibility.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This resolves #17774.
This ticket is because the fixes in GH #17154 failed to get every case,
leaving this one outlier to be fixed by this commit.
The text in https://github.com/Perl/perl5/issues/17154 gives extensive
details as to the problem. But briefly, in an attempt to speed up
interpreter cloning, I moved certain SVs from interpreter level to
global level in e80a0113c4a8036dfb22aec44d0a9feb65d36fed (v5.27.11,
March 2018). This was doable, we thought, because the content of these
SVs is constant throughout the life of the program, so no need to copy
them when cloning a new interpreter or thread. However when an
interpreter exits, all its SVs get cleaned up, which caused these to
become garbage in applications where another interpreter remains
running. This circumstance is rare enough that the bug wasn't reported
until September 2019, #17154. I made an initial attempt to fix the
problem, and closed that ticket, but I overlooked one of the variables,
which was reported in #17774, which this commit addresses.
Effectively the behavior is reverted to the way it was before
e80a0113c4a8036dfb22aec44d0a9feb65d36fed.
|
| |
|
|
|
|
|
|
|
|
| |
This makes special-cased forms such as sort { $b <=> $a }
even faster.
Also, since this commit removes PL_sort_RealCmp, it fixes the
issue with nested sort calls mentioned in gh #16129
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit enhances these functions so that on threaded perls, they use
mbrtowc and wcrtomb when available, making them thread safe. The
substitution isn't completely transparent, as no effort is made to hide
any differences in errno setting upon error. And there may be slight
differences in edge case behavior on some platforms.
This commit also changes the behaviors so that they take a scalar
parameter instead of a char *, and this might be 'undef' or not be
forceable into a valid PV. If not a PV, the functions initialize the
shift state. Previously the shift state was always reinitialized with
every call, which meant these could not work on locales with shift
states.
In addition, there were several issues in mbtowc and wctomb that this
commit fixes.
mbtowc and wctomb, when used, are now run with a semaphore. This avoids
races if called at the same time in another thread.
The returned wide character from mbtowc() could well have been garbage.
The final parameter to mbtowc is now optional, as passing an SV allows
us to determine the length without the need for an extra parameter. It
is now used only to restrict the parsing of the string to shorter than
the actual length.
wctomb would segfault if the string parameter was shared or hadn't
been pre-allocated with a string of sufficient length to hold the
result.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit changes the behavior so that it takes a scalar parameter
instead of a char *, and thus might not be forceable into a valid PV.
When not a PV, the shift state is reinitialized, like calling mblen with
a NULL first parameter. Previously the shift state was always
reinitialized with every call, which meant this could not work on
locales with shift states.
This commit also changes to use mbrlen() on threaded perls transparently
(mostly), when available, to achieve thread-safe operation. It is not
completely transparent because mbrlen (under the very rare stateful
locales) returns a different value when it's resetting the shift state.
It also may set errno differently upon errors, and no effort is made to
hide that difference. Also mbrlen on some platforms can handle partial
characters.
[perl #133928] showed that someone was having trouble with shift states.
|
|
|
|
|
| |
and the associated commits, at least until a way to make
wrap_op_checker() work is available.
|
|
|
|
| |
Fixes issue #14816
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is part of fixing gh #17154
This scenario from the ticket
(https://github.com/Perl/perl5/issues/17154#issuecomment-558877358)
shows why this fix is necessary:
main interpreter initializes PL_AboveLatin1 to an SV it owns
loads threads::lite and creates a new thread/interpreter which
initializes PL_AboveLatin1 to a SV owned by the new interpreter
threads::lite child interpreter finishes, freeing all of its SVs,
PL_AboveLatin1 is now invalid
main interpreter uses a regexp that relies on PL_AboveLatin1, dies
horribly.
By making these interpreter level variables, this is avoided. There is
extra copying, but it is just the SV headers, as the real data is kept
as static C arrays.
|
|
|
|
|
|
|
| |
Currently, whether the OS-level signal handler function is declared as
1-arg or 3-arg depends on the configuration. Add explicit versions of
these functions, principally so that POSIX.xs can call which version of
the handler it wants regardless of configuration: see next commit.
|
|
|
|
|
|
| |
With the revamping done in cc288b7a2732c37504039083ebb98241954636be, the
table of Unicode case folds that are more than a single character is no
longer used, so no need to generate it, or having it available.
|
|
|
|
| |
Also references to the term.
|
|
|
|
| |
This is part of this branch of changes.
|
|
|
|
| |
These were missed by 059703b088f44d5665f67fba0b9d80cad89085fd.
|
|
|
|
|
|
|
|
|
|
| |
I am starting to write a Unicode::Private_Use module which will allow
one to specify the Unicode properties of private use code points, thus
making them actually useful. This commit adds a hook to regcomp.c to
accommodate this module. The changes are pretty minimal. This way we
don't have to wait another release cycle to get it out there.
I don't want to document this interface, until it's proven.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The MY_CXT subsystem allows per-thread pseudo-static data storage.
Part of the implementation for this involves each XS module being
assigned a unique index in its my_cxt_index static var when first
loaded.
Because PERL_GLOBAL_STRUCT bans any static vars, under those builds
there is instead a table which maps the MY_CXT_KEY identifying string to
index.
Unfortunately, this table was allocated per-interpreter rather than
globally, meaning if multiple threads tried to load the same XS module,
crashes could ensue.
This manifested itself in failures in
ext/XS-APItest/t/keyword_plugin_threads.t
The fix is relatively straightforward: allocate PL_my_cxt_keys globally
rather than per-interpreter.
Also record the size of this struct in a new var, PL_my_cxt_keys_size,
rather than doing double duty on PL_my_cxt_size.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fix the various Perl_PerlSock_dup2_cloexec() type functions so that
t/porting/liberl.a passes under -DPERL_GLOBAL_STRUCT_PRIVATE builds.
In these builds it is forbidden to have any static variables, but each
of these functions (via convoluted macros) has a static var called
'strategy' which records, for each function, whether a run-time probe
has been done to determine the best way of achieving close-exec
functionality, and the result.
Replace them all with 'global' vars: PL_strategy_dup2 etc.
NB these vars aren't thread-safe but it doesn't really matter, as the
worst that can happen is for a redundant probe or two to be done before
a suitable "don't probe any more" value is written to the var and seen
by all the threads.
|
|
|
|
|
|
|
| |
A global hash has to be specially handled. The keys can't be shared,
and all the SVs stored into it must be in its thread. This commit adds
the hash, and initialization, and macros for context change, but doesn't
use them. The code to deal with this is entirely confined to regcomp.c.
|
|
|
|
| |
This will be used in future commits
|
|
|
|
| |
It currently is always set false, until later in this series of commits.
|
|
|
|
| |
This will be used in a future commit.
|
|
|
|
|
| |
The inversion list this refers to now includes the Latin 1 range, so the
name was misleading.
|
|
|
|
|
|
| |
This variable's name was out-of-date and misleading. It is the name of
an inversion list that contains all the code points in the current
version of Unicode that participate in any way in a /i type of fold.
|
|
|
|
|
|
|
| |
This table contains all the code points that are in any multi-character
fold (not the folded-from character, but what that character folds to).
It will be used in a future commit.
|
|
|
|
|
| |
These variables are constant, once initialized, through the life of a
program, so having them be per instance is a waste of time and space
|
|
|
|
|
|
|
|
|
|
|
|
| |
Under /a pattern matching, the matches of the [:posix:] classes are
restricted to the ASCII range. Previously, in a time/space trade-off
that favored space, we created the list of matching characters at
pattern compilation time by ANDing the full-range Posix class with the
set of ASCII characters.
But now, the tables for just the ASCII-range classes are generated
anyway, so there's no need to do that compilation-time intersection.
This slightly simplifies the code.
|
|
|
|
|
|
|
|
|
|
| |
This commit changes to use the C data structures generated by the
previous commit to compute what characters fold to a given one. This is
used to find out what things should match under /i.
This now avoids the expensive start up cost of switching to perl
utf8_heavy.pl, loading a file from disk, and constructing a hash from
it.
|
|
|
|
|
| |
These were for when some of the Posix character classes were implemented
as swashes, which is no longer the case, so these can be removed.
|
|
|
|
|
|
|
|
| |
This commit makes the inversion lists for parsing character name global
instead of interpreter level, so can be initialized once per process,
and no copies are created upon new thread instantiation. More
importantly, this is another instance where utf8_heavy.pl no longer
needs to be loaded, and the definition files read from disk.
|
|
|
|
|
| |
These are now constant through the life of the program, so don't need to
be duplicated at each new thread instantiation.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Prior to this commit, if a program wanted to compute the case-change of
a character above 0xFF, the C code would switch to perl, loading
lib/utf8heavy.pl and then read another file from disk, and then create a
hash. Future references would use the hash, but the start up cost is
quite large. There are five case change types, uc, lc, tc, fc, and
simple fc. Only the first encountered requires loading of utf8_heavy,
but each required switching to utf8_heavy, and reading the appropriate
file from disk.
This commit changes these functions to use compiled-in C data structures
(inversion maps) to represent the data. To look something up requires a
binary search instead of a hash lookup.
An individual hash lookup tends to be faster than a binary search, but
the differences are small for small sizes. I did some benchmarking some
years ago, (commit message 87367d5f9dc9bbf7db1a6cf87820cea76571bf1a) and
the results were that for fewer than 512 entries, the binary search was
just as fast as a hash, if not actually faster. Now, I've done some
more benchmarks on blead, using the tool benchmark.pl, which wasn't
available back then. The results below indicate that the differences
are minimal up through 2047 entries, which all Unicode properties are
well within.
A hash, PL_foldclosures, is still constructed at runtime for the case of
regular expression /i matching, and this could be generated at Perl
compile time, as a further enhancement for later. But reading a file
from disk is no longer required to do this.
======================= benchmarking results =======================
Key:
Ir Instruction read
Dr Data read
Dw Data write
COND conditional branches
IND indirect branches
_m branch predict miss
_m1 level 1 cache miss
_mm last cache (e.g. L3) miss
- indeterminate percentage (e.g. 1/0)
The numbers represent raw counts per loop iteration.
"\x{10000}" =~ qr/\p{CWKCF}/"
swash invlist Ratio %
fetch search
------ ------- -------
Ir 2259.0 2264.0 99.8
Dr 665.0 664.0 100.2
Dw 406.0 404.0 100.5
COND 406.0 405.0 100.2
IND 17.0 15.0 113.3
COND_m 8.0 8.0 100.0
IND_m 4.0 4.0 100.0
Ir_m1 8.9 17.0 52.4
Dr_m1 4.5 3.4 132.4
Dw_m1 1.9 1.2 158.3
Ir_mm 0.0 0.0 100.0
Dr_mm 0.0 0.0 100.0
Dw_mm 0.0 0.0 100.0
These were constructed by using the file whose contents are below, which
uses the property in Unicode that currently has the largest number of
entries in its inversion list, > 1600. The test was run on blead -O2,
no debugging, no threads. Then the cut-off boundary was changed from
512 to 2047 for when we use a hash vs an inversion list, and the test
run again. This yields the difference between a hash fetch and an
inversion list binary search
===================== The benchmark file is below ===============
no warnings 'once';
my @benchmarks;
push @benchmarks, 'swash' => {
desc => '"\x{10000}" =~ qr/\p{CWKCF}/"',
setup => 'no warnings "once"; my $re = qr/\p{CWKCF}/; my $a =
"\x{10000}";',
code => '$a =~ $re;',
};
\@benchmarks;
|
|
|
|
|
|
|
|
|
|
| |
These structures are read-only, use const C strings, and are truly
global, so no need to have them be interpreter level. This saves
duplicating and freeing them as threads come and go.
In doing this, I noticed that not every one was properly being
copied/deallocated, so this fixes some potential unreported bugs, and
leaks.
|
|
|
|
|
|
| |
This (large) commit allows locales to be used in threaded perls on
platforms that support it. This includes recent Windows and Posix 2008
ones.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It is possible for operations on threaded perls which don't 'use locale'
to still change the locale. This happens when calling
POSIX::localeconv() and I18N::Langinfo(), and in earlier perls, it can
happen for other operations when perl has been initialized with the
environment causing the various locale categories to not have a uniform
locale.
This commit causes the areas where the locale for this category should
predictably be in one or the other state to be a critical section where
another thread can't interrupt and change it. This is a separate
mutex, so that only these particular operations will be held up.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
khw could not find any modules on CPAN that correctly use the C library
function setlocale(). (The very few that do try, do not use it
correctly, looking at the return value incorrectly, so they are broken.)
This analysis does not include modules that call non-Perl libaries that
may call setlocale().
And, a future commit will render the setlocale() function useless in
some configurations on some platforms.
So this commit adds Perl_setlocale(), for XS code to call, and which is
always effective, but it should not be used to alter the locale except
on platforms where the predefined variable ${^SAFE_LOCALES} evaluates to
1.
This function is also what POSIX::setlocale() calls to do the real work.
|
|
|
|
|
|
|
|
|
| |
On systems that have the POSIX 2008 operations, including
nl_langinfo_l(), this commit causes them to not have to actually change
the locale when determining what the decimal point character is.
The locale may have to change during the printing/reading of numbers,
but eventually we can use sprintf_l(), if available, to avoid that too.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Some locales are UTF-8, some are not. Knowledge of this is needed in
various circumstances. This commit saves the results of the last
several lookups so they don't have to be recalculated each time.
The full generality of POSIX locales is such that you can have error
messages be displayed in one locale, say Spanish, while other things are
in French. To accommodate this generality, the program can loop through
all the locale categories finding the UTF8ness of the locale it points
to. However, in almost all instances, people are going to be in either
French or in Spanish, and not in some combination. Suppose it is a
French UTF-8 locale for all categories. This new cache will know that
the French locale is UTF-8, and the queries for all but the first
category can return that immediately.
This simple cache avoids the overhead of hashes.
This also fixes a bug I realized exists in threaded perls, but haven't
reproduced. We do not support locales in such perls, and the user must
not change the locale or 'use locale'. But perl itself could change the
locale behind the scenes, leading to segfaults or incorrect results.
One such instance is the determination of UTF8ness. But this only could
happen if the full generality of locales is used so that the categories
are not all in the same locale. This could only happen (if the user
doesn't change locales) if the environment is such that the perl program
is started up so that the categories are in such a state. This commit
fixes this potential bug by caching the UTF8ness of each category at
startup, before any threads are instantiated, and so checking for it
later just looks it up in the cache, without perl changing the locale.
|
|
|
|
|
|
|
|
|
|
| |
The LC_NUMERIC locale category is kept so that generally the decimal
point (radix) is a dot. For some (mostly) output purposes, it needs to
be swapped into the program's current underlying locale so that a
non-dot can be printed.
This commit changes things so that if the current underlying locale uses
a decimal point, the swap doesn't happen, as it's not needed.
|
|
|
|
| |
This somehow became unused or never got used; I didn't do the research.
|
|
|
|
| |
As explained in the docs, this helps detect spoofing attacks.
|