| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
| |
plus some typo fixes. I probably changed some things in perlintern, too.
|
|
|
|
|
|
|
| |
In those cases where the hash key comes from a hek, we already have a
computed hash value, so pass that to hv_common.
The easiest way to accomplish this is to add a new macro.
|
|
|
|
|
| |
This uses macros which work cross-platform. This has the added advantge
what is going on is much clearer.
|
|
|
|
|
|
|
| |
Iterated hashes shouldn’t have to allocate space for something
specific to stashes, so move the SUPER method cache from the
HvAUX struct (which all iterated hashes have) into the mro
meta struct (which only stashes have).
|
|
|
|
|
|
|
| |
This is left over from when READONLY+FAKE meant copy-on-write.
Read-only copy-on-write scalars (which could not occur with the old
way of flagging things) must not be exempt from hash key restrictions.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is part of #116907, too. It also fixes #72924 as a side effect;
the next commit will explain.
The value of pos($foo) was being stored as an I32, not allowing values
above I32_MAX. Change it to SSize_t (the signed equivalent of size_t,
representing the maximum string length the OS/compiler supports).
This is accomplished by changing the size of the entry in the magic
struct, which is the simplest fix.
Other parts of the code base can benefit from this, too.
We actually cast the pos value to STRLEN (size_t) when reading
it, to allow *very* long strings. Only the value -1 is special,
meaning there is no pos. So the maximum supported offset is
2**sizeof(size_t)-2.
The regexp engine itself still cannot handle large strings, so being
able to set pos to large values is useless right now. This is but one
piece in a larger puzzle.
Changing the size of mg->mg_len also requires that
Perl_hv_placeholders_p change its type. This function
should in fact not be in the API, since it exists
solely to implement the HvPLACEHOLDERS macro. See
<https://rt.perl.org/rt3/Ticket/Display.html?id=116907#txn-1237043>.
|
|
|
|
| |
It was not clear that it referred to uvar magic.
|
|
|
|
|
|
| |
This avoids HvFILL() being O(n) for large n on large hashes, but also avoids
storing the value of HvFILL() in smaller hashes (ie a memory overhead on
every single object built using a hash.)
|
|
|
|
|
|
|
| |
No keys implies no chains used, so the return value is 0. One key
unambiguously means 1 chain used, and all the others are free. Two or more
keys might share the same chain, or might not, so the calculation can't be
short-circuited.
|
|
|
|
|
| |
The are lots of places where local vars aren't used when compiled
with NO_TAINT_SUPPORT.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds support for PERL_PERTURB_KEYS environment variable, which in turn allows one to control
the level of randomization applied to keys() and friends.
When PERL_PERTURB_KEYS is 0 we will not randomize key order at all. The
chance that keys() changes due to an insert will be the same as in
previous perls, basically only when the bucket size is changed.
When PERL_PERTURB_KEYS is 1 we will randomize keys in a non repeatedable
way. The chance that keys() changes due to an insert will be very high.
This is the most secure and default mode.
When PERL_PERTURB_KEYS is 2 we will randomize keys in a repeatedable way.
Repititive runs of the same program should produce the same output every
time. The chance that keys changes due to an insert will be very high.
This patch also makes PERL_HASH_SEED imply a non-default
PERL_PERTURB_KEYS setting. Setting PERL_HASH_SEED=0 (exactly one 0) implies
PERL_PERTURB_KEYS=0 (hash key randomization disabled), settng PERL_HASH_SEED
to any other value, implies PERL_PERTURB_KEYS=2 (deterministic/repeatable
hash key randomization). Specifying PERL_PERTURB_KEYS explicitly to a
different level overrides this behavior.
Includes changes to allow one to compile out various aspects of the
patch. One can compile such that PERL_PERTURB_KEYS is not respected, or
can compile without hash key traversal randomization at all. Note that
support for these modes is incomplete, and currently a few tests will
fail.
Also includes a new subroutine in Hash::Util::hash_traversal_mask()
which can be used to ensure a given hash produces a predictable key
order (assuming the same hash seed is in effect). This sub acts as a
getter and a setter.
NOTE - this patch lacks tests, but I lack tuits to get them done quickly,
so I am pushing this with the hope that others can add them afterwards.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The usages are as far as I know incorrect anyway. We resize
the hash bucket array based on the number of keys it holds,
not based on the number of buckets that are used, so this
usage was wrong anyway.
Another bug that this revealed is that the old code would allow
HvMAX(hv) to fall to 0, even though every other part of the
core expects it to have a minimum of 7 (meaning 8 buckets).
As part of this we change the hard coded 7 to a defined constant
PERL_HASH_DEFAULT_HvMAX.
After this patch there remains one use of HvFILL in core, that used
for scalar(%hash) which I plan to remove in a later patch.
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Inserting into a hash that is being traversed with each()
has always produced undefined behavior. With hash traversal
randomization this is more pronounced, and at the same
time relatively easy to spot. At the cost of an extra U32
in the xpvhv_aux structure we can detect that the xhv_rand
has changed and then produce a warning if it has.
It was suggested on IRC that this should produce a fatal
error, but I couldn't see a clean way to manage that with
"strict", it was much easier to create a "severe" (internal)
warning, which is enabled by default but suppressible with
C<no warnings "internal";> if people /really/ wanted.
|
|
|
|
|
|
|
| |
This serves two functions, it makes it harder for an attacker
to learn useful information by viewing the output of keys(),
and it makes "insert during traversal" errors much easier to
spot, as they will almost always produce degenerate behavior.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When inserting into a hash results in a collision the order of the items
in the bucket chain is predictable (FILO), and can be used to determine
that a collision has occured.
When a hash is too small for the number of items it holds we double
its size and remap the items as required. During this process the
keys in a bucket will reverse order, and exposes information to an
attacker that a collision has occured.
We therefore use the PL_hash_rand_bits() and the S_ptr_hash()
infrastructure to randomly "perturb" the order that colliding
items are inserted into the bucket chain. During insertion and
mapping instead of doing a simple "insert to top" we check the low
bit of PL_hash_rand_bits() and depending if it is set or not we
insert at the top of the chain, otherwise second from the top.
The end result being that the order in a bucket is less predictable,
which should make it harder for an attacker to spot a collision.
Every insert (via hv_common), and bucket doubling (via hsplit())
results in us updating PL_hash_rand_bits() using "randomish" data
like the hashed bucket address, the hash of the inserted item, and
the address of the inserted item.
This also updates the xhv_rand() of the hash, if there is one, during
S_hsplit() so that the iteration order changes when S_hsplit() is
called. This also is intended to make it harder for an attacker to
aquire information about collisions.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds:
S_ptr_hash() - A new static function in hv.c which can be used to
hash a pointer or integer.
PL_hash_rand_bits - A new interpreter variable used as a cheap
provider of "semi-random" state for use by the hash infrastructure.
xpvhv_aux.xhv_rand - Used as a mask which is xored against the
xpvhv_aux.riter during iteration to randomize the order the actual
buckets are visited.
PL_hash_rand_bits is initialized as interpreter start from the random
hash seed, and then modified by "mixing in" the result of ptr_hash()
on the bucket array pointer in the hv (HvARRAY(hv)) every time
hv_auxinit() allocates a new iterator structure.
The net result is that every hash has its own iteration order, which
should make it much more difficult to determine what the current hash
seed is.
This required some test to be restructured, as they tested for something
that was not necessarily true, we never guaranteed that two hashes with
the same keys would produce the same key order, we merely promised that
using keys(), values(), or each() on the same hash, without any
insertions in between, would produce the same order of visiting the
key/values.
|
|
|
|
| |
This saves one call to HvPLACEHOLDERS_get().
|
|
|
|
|
|
|
|
|
|
|
| |
Which makes me realise that if we clear placeholders, we may be able to
avoid the need to split at all.
(In fact, as hv_common() only adds one key to the hash, under the current
definition of DO_HSPLIT() which only considers total number of keys,
clearing any placeholder is going to be enough to drop the total number of
keys, and so no longer trigger the split. But we'll leave the code making a
second check, to avoid a tight coupling with the internals of DO_HSPLIT().)
|
|
|
|
|
| |
Seems pointless to check the exit condition before any iterations, when we
know that it will always be false the first time.
|
|
|
|
|
| |
The code duplication that introduced hv_ksplit() as a fork of hsplit() back
with commit 72940dca186befa0 in Sept 1996 is finally healed.
|
|
|
|
|
|
|
| |
This mimics the behaviour in Perl_hv_ksplit().
Also remove a vestigial comment. The code it relates to was removed in
commit 7dc8663964c66a69 in Nov 2012.
|
|
|
|
|
|
|
|
| |
Whilst this is slightly more work for its existing two callers, it will
permit Perl_hv_ksplit() to also call it.
Use STRLEN for the parameters, and change a local variable from I32 to
STRLEN to match.
|
|
|
|
|
| |
This makes the rest of the code of Perl_hv_ksplit() closer to that of
S_hsplit().
|
|
|
|
| |
Making the code as similar as possible will make it simpler to merge the two.
|
|
|
|
|
|
|
|
|
|
| |
The relevant code calls Perl_hv_clear_placeholders() at split time, if there
are still placeholders left over from a (previously) restricted hash.
There are two callers to S_hsplit(), one from the regular HV code, and one
from the shared string table code. As the shared string table can't contain
placeholders, only the other call site could trigger this condition, so move
the code there. This simplifies S_hsplit(), and will make splitting the
shared string table marginally faster.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
STRANGE_MALLOC was added in 5.002 beta 1 (4633a7c4bad06b47) as part of an
work around for typical mallocs, which had a bad interaction with perl's
allocation needs. Specifically, repeatedly extending an array and then
creating SV heads (such as when reading lines of a file into an array)
could end up with each reallocation for the array being unable to extend in
place, needing a fresh chunk of memory, and the released memory not being
suitable for use as more SV heads, so sitting unused. The solution was for
perl to recycle the old array body as SV heads, instead of returning it to
the system, passing the memory from the the AV code to the SV code using
offer_nice_chunk(), PL_nice_chunk and PL_nice_chunk_size.
STRANGE_MALLOC was actually a signal that the malloc() didn't need
protecting from itself, and to disable the work around.
offer_nice_chunk(), PL_nice_chunk and PL_nice_chunk_size were removed by
commit 9a87bd09eea1d037 in Nov 2010, without any ill effects, hence the
code used when STRANGE_MALLOC was *not* defined is essentially doing extra
work for no benefits.
From the lack of problems reported, one can assume that in the intervening
15 years malloc technology has got significantly improved, and it is probably
better to be honest with it, rather than trying to second guess it.
Hence remove all the non-STRANGE_MALLOC code, and leave everyone using the
much simpler code. See also
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-11/msg00495.html
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2013-01/msg00126.html
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The purpose is less machine instructions/faster code.
* S_hv_free_ent_ret() is always called with entry non-null: so change its
signature to reflect this, and remove a null check;
* Add some SvREFCNT_dec_NNs;
* In hv_clear(), refactor the code slightly to only do a SvREFCNT_dec_NN
within the branch where its already been determined that the arg is
non-null; also, use the _nocontext variant of Perl_croak() to save
a push instruction in threaded perls.
|
|
|
|
|
| |
In this instance, we know that av is not null, so no need to check
whether it is
|
|
|
|
|
|
|
| |
There is no point in modifying a struct just before freeing it.
This was my mistake, in commit 47f1cf7702, when I moved the code from
S_hfreeentries to hv_undef.
|
|
|
|
|
|
|
| |
This finishes the removal of register declarations started by
eb578fdb5569b91c28466a4d1939e381ff6ceaf4. It neglected the ones in
function parameter declarations, and didn't include things in dist, ext,
and lib, which this does include
|
|
|
|
|
|
| |
We do not want to resize the hash every time the bucket length is
too long. Nor do we want to pay the price of checking how long
the bucket length is when there is nothing we can do about it anyway.
|
|
|
|
|
| |
note: this failed to build in smoke-me eg.
http://perl.develop-help.com/raw/?id=131750
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This patch does the following:
*) Introduces multiple new hash functions to choose from at build
time. This includes Murmur-32, SDBM, DJB2, SipHash, SuperFast, and
One-at-a-time. Currently this is handled by muning hv.h. Configure
support hopefully to follow.
*) Changes the default hash to Murmur hash which is faster than the
old default One-at-a-time.
*) Rips out the old HvREHASH mechanism and replaces it with a
per-process random hash seed.
*) Changes the old PL_hash_seed from an interpreter value to a
global variable. This means it does not have to be copied during
interpreter setup or cloning.
*) Changes the format of the PERL_HASH_SEED variable to a hex
string so that hash seeds longer than fit in an integer are possible.
*) Changes the return of Hash::Util::hash_seed() from a number to a
string. This is to accomodate hash functions which have more bits than
can be fit in an integer.
*) Adds new functions to Hash::Util to improve introspection of hashes
-) hash_value() - returns an integer hash value for a given string.
-) bucket_info() - returns basic hash bucket utilization info
-) bucket_stats() - returns more hash bucket utilization info
-) bucket_array() - which keys are in which buckets in a hash
More details on the new hash functions can be found below:
Murmur Hash: (v3) from google, see
http://code.google.com/p/smhasher/wiki/MurmurHash3
Superfast Hash: From Paul Hsieh.
http://www.azillionmonkeys.com/qed/hash.html
DJB2: a hash function from Daniel Bernstein
http://www.cse.yorku.ca/~oz/hash.html
SDBM: a hash function sdbm.
http://www.cse.yorku.ca/~oz/hash.html
SipHash: by Jean-Philippe Aumasson and Daniel J. Bernstein.
https://www.131002.net/siphash/
They have all be converted into Perl's ugly macro format.
I have not done any rigorous testing to make sure this conversion
is correct. They seem to function as expected however.
All of them use the random hash seed.
You can force the use of a given function by defining one of
PERL_HASH_FUNC_MURMUR
PERL_HASH_FUNC_SUPERFAST
PERL_HASH_FUNC_DJB2
PERL_HASH_FUNC_SDBM
PERL_HASH_FUNC_ONE_AT_A_TIME
Setting the environment variable PERL_HASH_SEED_DEBUG to 1 will make
perl output the current seed (changed to hex) and the hash function
it has been built with.
Setting the environment variable PERL_HASH_SEED to a hex value will
cause that value to be used at the seed. Any missing bits of the seed
will be set to 0. The bits are filled in from left to right, not
the traditional right to left so setting it to FE results in a seed
value of "FE000000" not "000000FE".
Note that we do the hash seed initialization in perl_construct().
Doing it via perl_alloc() (via init_tls) causes problems under
threaded builds as the buffers used for reentrant srand48 functions
are not allocated. See also the p5p mail "Hash improvements blocker:
portable random code that doesnt depend on a functional interpreter",
Message-ID:
<CANgJU+X+wNayjsNOpKRqYHnEy_+B9UH_2irRA5O3ZmcYGAAZFQ@mail.gmail.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
By defining NO_TAINT_SUPPORT, all the various checks that perl does for
tainting become no-ops. It's not an entirely complete change: it doesn't
attempt to remove the taint-related interpreter variables, but instead
virtually eliminates access to it.
Why, you ask? Because it appears to speed up perl's run-time
significantly by avoiding various "are we running under taint" checks
and the like.
This change is not in a state to go into blead yet. The actual way I
implemented it might raise some (valid) objections. Basically, I
replaced all uses of the global taint variables (but not PL_taint_warn!)
with an extra layer of get/set macros (TAINT_get/TAINTING_get).
Furthermore, the change is not complete:
- PL_taint_warn would likely deserve the same treatment.
- Obviously, tests fail. We have tests for -t/-T
- Right now, I added a Perl warn() on startup when -t/-T are detected
but the perl was not compiled support it. It might be argued that it
should be silently ignored! Needs some thinking.
- Code quality concerns - needs review.
- Configure support required.
- Needs thinking: How does this tie in with CPAN XS modules that use
PL_taint and friends? It's easy to backport the new macros via PPPort,
but that doesn't magically change all code out there. Might be
harmless, though, because whenever you're running under
NO_TAINT_SUPPORT, any check of PL_taint/etc is going to come up false.
Thus, the only CPAN code that SHOULD be adversely affected is code
that changes taint state.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
I don't see any reason not to set flags properly in this
branch. It doesn't look like any useful optimization.
It's probably even a bug, but probably it can only be hit from
a XS code. To hit the bug keysv should be provided, be UTF8
and not SvIsCOW_shared_hash, but with flags containing
HVhek_KEYCANONICAL.
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
When %^H is copied on entering a new scope, if it happens to have been
tied it can die. This was resulting in leaks, because no protections
were added to handle that case.
The two things that were leaking were the new hash in hv_copy_hints_hv
and the new value (for an element) in newSVsv.
By fixing newSVsv itself, this also fixes any potential leaks when
other pieces of code call newSVsv on explosive values.
|
|
|
|
|
|
|
|
| |
The current iterator was leaking when a tied hash was freed or
undefined.
Since we already have a mechanism, namely HvLAZYDEL, for freeing
HvEITER when not referenced elsewhere, we can use that.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Perl caches SUPER methods inside packages named Foo::SUPER. But this
interferes with actual method calls on those packages (SUPER->foo,
foo::SUPER->foo).
The first time a package is looked up, it is vivified under the name
with which it is looked up. So *SUPER:: will cause that package
to be called SUPER, and *main::SUPER:: will cause it to be named
main::SUPER.
main->SUPER::isa used to be very sensitive to the name of the
main::FOO package (where the cache is kept). If it happened to be
called SUPER, that call would fail.
Fixing that bug (commit 3c104e59d83f) caused the CPAN module named
SUPER to fail, because SUPER->foo was now being treated as a
SUPER::method call. gv_fetchmeth_pvn was using the ::SUPER suffix to
determine where to look for the method. The package passed to it (the
::SUPER package) was being used to look for cached methods, but the
package with ::SUPER stripped off was being used for the rest of
lookup. 3c104e59d83f made main->SUPER::foo work by treating SUPER
as main::SUPER in that case. Mentioning *main::SUPER:: or doing a
main->SUPER::foo call before loading SUPER.pm also caused it to fail,
even before 3c104e59d83f.
Instead of using publicly-visible packages for internal caches, we
should be keeping them internal, to avoid such side effects.
This commit adds a new member to the HvAUX struct, where a hash of GVs
is stored, to cache super methods. I cannot simpy use a hash of CVs,
because I need GvCVGEN. Using a hash of GVs allows the existing
method cache code to be used.
This new hash of GVs is not actually a stash, as it has no HvAUX
struct (i.e., no name, no mro_meta). It doesn’t even need an @ISA
entry as before (which was only used to make isa caches reset), as it
shares its owner stash’s mro_meta generation numbers. In fact, the
GVs inside it have their GvSTASH pointers pointing to the owner stash.
In terms of memory use, it is probably the same as before. Every
stash and every iterated or weakly-referenced hash is now one pointer
larger than before, but every SUPER cache is smaller (no HvAUX, no
*ISA + @ISA + $ISA[0] + magic).
The code is a lot simpler now and uses fewer stash lookups, so it
should be faster.
This will break any XS code that expects the gv_fetchmeth_pvn to treat
the ::SUPER suffix as magical. This behaviour was only barely docu-
mented (the suffix was mentioned, but what it did was not), and is
unused on CPAN.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This removes most register declarations in C code (and accompanying
documentation) in the Perl core. Retained are those in the ext
directory, Configure, and those that are associated with assembly
language.
See:
http://stackoverflow.com/questions/314994/whats-a-good-example-of-register-variable-usage-in-c
which says, in part:
There is no good example of register usage when using modern compilers
(read: last 10+ years) because it almost never does any good and can do
some bad. When you use register, you are telling the compiler "I know
how to optimize my code better than you do" which is almost never the
case. One of three things can happen when you use register:
The compiler ignores it, this is most likely. In this case the only
harm is that you cannot take the address of the variable in the
code.
The compiler honors your request and as a result the code runs slower.
The compiler honors your request and the code runs faster, this is the least likely scenario.
Even if one compiler produces better code when you use register, there
is no reason to believe another will do the same. If you have some
critical code that the compiler is not optimizing well enough your best
bet is probably to use assembler for that part anyway but of course do
the appropriate profiling to verify the generated code is really a
problem first.
|
|
|
|
|
| |
I was confused by the earlier documentation. Thanks to Leon Timmermans
for clarifying, and to Vicent Pitt for most of the wording
|
|
|
|
|
| |
This updates the editor hints in our files for Emacs and vim to request
that tabs be inserted as spaces.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Commit 104d7b69 made sv_clear free hashes iteratively rather than recursively;
however, my code didn't record the current hash index when freeing a
nested hash, which made the code go quadratic when freeing a large hash
with inner hashes, e.g.:
my $r; $r->{$_} = { a => 1 } for 1..10_0000;
This was noticeable on such things as CPAN.pm being very slow to exit.
This commit fixes this by squirrelling away the old hash index in the
now-unused SvMAGIC field of the hash being freed.
|
| |
|
|
|
|
|
|
|
| |
Commit 60edcf09a was supposed to make things less buggy, but putting
the ENTER/LEAVE in h_freeentries was a mistake, as both hv_undef and
hv_clear still access the hv after calling h_freeentries. Why it
didn’t crash for me is anyone’s guess.
|