| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
|
|
|
|
|
| |
The state of porting to EBCDIC that's going to get into 5.22 leaves many
many error lines being displayed when 'make test' is run if the /cpan
directory is included. Reluctantly, I'm excluding these tests for 5.22.
I plan to work on either fixing or having individual skips in them in
5.23, but that is not the current state.
|
| |
|
|
|
| |
This works around a problem with z/OS where if there are unread lines
from a file descriptor using this, a SIGPIPE error is raised.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
See ticket #123561. The presence of ‘glob’ was causing perl to
attempt to File::Glob at compile time, before @INC is set up.
Windows needs to do file globbing on @ARGV for t/TEST to work cor-
rectly. Windows only uses TEST for miniperl (it uses harness for
‘make test’), and under miniperl ‘glob’ doesn’t use File::Glob. So it
is safe to use ‘glob’ on Windows. For the sake of other systems, we
can put it in a string eval to avoid even compiling the op.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
since Win32 perl doesn't glob by default
at this point the following tests fail:
io/dup.t
io/fs.t
io/open.t
io/perlio_leaks.t
op/coreamp.t
op/filetest.t
op/fork.t
op/fresh_perl_utf8.t
op/glob.t
op/heredoc.t
op/magic.t
op/read.t
op/readline.t
op/sselect.t
op/stat.t
op/substr.t
op/tie_fetch_count.t
op/write.t
|
| | |
|
| |
|
|
|
|
|
|
| |
This reverts commit da6e3e61d3d3194e7f769b7ddf7e6d16904c24d3.
This causes problems for deparse mode, which emits pages of warnings.
If we are going to add -w, we ought first to fix deparse testing to
work with it.
|
| |
|
|
| |
This reverts commit 448bc2e91e255b2610f23d9fe8092c9143dac28e.
|
| |
|
|
| |
This reverts commit 861a58daef615e7ade791b887911ef3caba8c539.
|
| |
|
|
| |
This reverts commit 3c8571535150a64cf1163a15106d0f9e57fdd6a7.
|
| |
|
|
| |
This reverts commit 861aa4195fddf02db2d2d1412c83a273daae9f05.
|
| |
|
|
| |
This reverts commit 922487b2e3c980db625829f57958aac6a3e33552.
|
| |
|
|
| |
This reverts commit 02a483e9608abd0f822d7c8ec6d64c07c4689e55.
|
| |
|
|
| |
This reverts commit b8059a1db50cef0ea798275144b6d2f54a288946.
|
| |
|
|
|
|
| |
description"
This reverts commit ba369d6f39f2f98f9e999dc978baeba4d4efba25.
|
| |
|
|
| |
This reverts commit e28ec392de528ba135a100709ce789a168286f86.
|
| |
|
|
| |
This reverts commit 08ac0af6a0260a3939c4904c299200adee71010d.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This patch creates an empty t/TestValgrind.pm file, and requires it
from t/TEST when ENV{PERL_VALGRIND} is set. As such its seeks to
reinterpret the comments at the top of t/TEST.
I propose that the "no require allowed" rule appply to code reached
from "make test" without options.
Code used only for special case testing, like "make test.valgrind", or
with options passed in via $ENV{TEST_OPTS}, need not be constrained
like this.
A broken require is not something seen until the statement is executed
and throws an error. There is nothing special about require at
parse-time that must be avoided, unless we're more than typically
paranoid.
make test.valgrind
/bin/sh -e ./runtests choose
./perl TEST
sh -c valgrind --log-fd=3 --suppressions=perl.supp --leak-check=yes --leak-resolution=high --show-reachable=yes --num-callers=50 --track-origins=yes ./perl -I.. -MTestInit porting/podcheck.t 3>podcheck.t.valgrind-current
valgrind --log-fd=3 --suppressions=perl.supp --leak-check=yes --leak-resolution=high --show-reachable=yes --num-callers=50 --track-origins=yes ./perl -I.. -MTestInit porting/podcheck.t
If this argument is accepted, it looks simple to move the VALGRIND
related code blocks into new subroutines in the now-empty
TestValgrind.pm file, and call them appropriately.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Recent addition of -w on shebang exposed 2 uninitialized vars, due to
VG_OPTS envar added in commit 11b1a7c9, which added glue to reuse
t/TEST's support for make test.valgrind to run linux perf. The 2
use-cases are:
1. make test.valgrind
2. export VG_OPTS='stat -o perf-stat --append --'
export VALGRIND=/usr/bin/perf
export VG_TEST=--help
make test.valgrind
Since 2 is rather non-obvious, clarify with comment here and in code.
TODO: this patch is old, and either needs to be revalidated, or
perhaps rethought; use-case 2 is mostly obsoleted by -lxperf option,
but case 1 still holds.
|
| |
|
|
|
|
|
| |
Commit 8e03ad8f6c wrote the tests' timing data to a Storable file
using 'perf' as the storage key, but now that we can capture real perf
stat data during make test, the old name is misleading - change it to
"times" to better describe its origin and character.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Support use of the --repeat=3 option of linux's perf stat. This
basically means extending the TAPv13 spec (or t/TEST's implementation)
to allow repeated (ie R) plan executions (ie R occurrences of "1..N"
plus the preceding or following N * "ok"s), as long as all R plans
have the same N.
When a test script is run (under linux perf with --repeat=N), the
value of R is unavailable in the output; it is transparent other than
the repetition itself. But we woundnt want it anyway; TAP readers
dont need to handle a new kind of input. If we just allow R
repetitions of 1..N tests, we get loose coupling and dont care if
--repeat=5 were given.
Current legal TAP is essentially either:
print "ok $_\n" for 1..N; # 1A
print "1..N\n";
# OR [1]
print "1..N\n"; # 1B
print "ok $_\n" for 1..N;
The big simplification here is that under --repeat, leading vs
trailing "1..N"s are an irrelevant detail.
- "1..N" plans are only allowed before any tests, or after all of
them, ie ($next==0 or $next==$plan). We just verify this when
"1..N" is seen in test output.
- $plan = undef initially, updated to N when "1..N" is 1st seen.
- $next=0 when "1..N" is seen [2].
- when 2nd+ "1..N" plans are seen, they are allowed if they agree on N.
- The change to if (defined $plan) accepts "1..0" as legal (ie SKIPs).
- $seen_leader and $trailing_leader are thus obsolete, replaced where
needed by "if ($plan)", which is set only when "1..N" is seen.
- $seen_ok was used with $seen_leader to reset $next (test-count).
now handled in [2]
- $max is also taken right from "1..N", replaced with <some-expr>($plan)
NOTES:
[1]: this patch allows BOTH 1A then 1B to run successfully, but not 1B
followed by 1A; the latter would fail on N+1 th test, because the
now-trailing "1..N" failed to reset $next. This is an unimportant
corner case and doesnt warrant cluttering details in the code.
We are running maintained regression tests here, not logic bombs.
- On "1..N" occurrences:
$Reps++ to count repeats (tested at end)
reset $next = 0 after validating it [1]
$nextreps++, like $next w/o reset.
- post-loop tests:
change ($next == $max) test to ($nextreps == $Reps * $plan),
which avoids the now resetting $next.
TODO:
This patch supports --repeat N in t/TEST only; harness is strict wrt
"only 1 plan", ala TAPv13, so make test_harness complains as follows,
and fails the tests.
comp/use.t ........................................................ All 84 subtests passed
comp/retainedlines.t .............................................. All 75 subtests passed
comp/cmdopt.t ..................................................... All 44 subtests passed
Sort whats reported in summary data; unique tests or repeated tests ?
Reduce $totmax vs $max vs $Reps a bit, extra/redundant state is not helpful.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With this patch, TEST can run each test-script under linux's perf
stat. Since previous patch honors options for $do_nothing / requires,
harness can use perf stat too.
t]$ ./perl -I../lib TEST -lxperf
t]$ ./perl harness -lxperf
"perf stat" output is comparable to that of "valgrind --tool=cachegrind"
but it takes very little extra CPU to run:
export TEST_OPTS=-lxperf
time make test time make test
-------------- --------------
real 25m41.122s real 26m30.719s
user 18m28.773s user 18m43.286s
sys 1m12.705s sys 1m33.398s
Given this small runtime cost, porters might be inclined to use it and
inspect the data. There may be some core regression tests that are
useful for benchmarking purposes.
EXAMPLES:
Performance counter stats for './perl base/cond.t':
2.077911 task-clock # 0.640 CPUs utilized
2 context-switches # 0.963 K/sec
1 cpu-migrations # 0.481 K/sec
519 page-faults # 0.250 M/sec
2,753,978 cycles # 1.325 GHz
1,725,265 stalled-cycles-frontend # 62.65% frontend cycles idle
2,405,122 stalled-cycles-backend # 87.33% backend cycles idle
3,278,429 instructions # 1.19 insns per cycle
# 0.73 stalled cycles per insn [52.13%]
<not counted> branches
<not counted> branch-misses
0.003244748 seconds time elapsed
perf stat --repeat N also computes std-deviation for each measurement
it does, so this patch makes that feature available. Std-dev will
tell us when performance differences are lost in the noise.
./perl TEST -v -lxperf=10 re/pat_rt_report.t
perf stat --repeat 10 -- ./perl -I.. -MTestInit re/pat_rt_report.t
./perl -I.. -MTestInit re/pat_rt_report.t
Performance counter stats for './perl -I.. -MTestInit re/pat_rt_report.t' (10 runs):
98.749506 task-clock # 0.001 CPUs utilized ( +- 23.68% )
68 context-switches # 0.686 K/sec ( +- 82.04% )
10 cpu-migrations # 0.097 K/sec ( +- 17.44% )
2,956 page-faults # 0.030 M/sec ( +- 23.22% )
254,477,657 cycles # 2.577 GHz ( +- 26.02% ) [50.43%]
75,556,485 stalled-cycles-frontend # 29.69% frontend cycles idle ( +- 19.50% ) [50.93%]
62,921,965 stalled-cycles-backend # 24.73% backend cycles idle ( +- 25.21% ) [51.50%]
241,522,257 instructions # 0.95 insns per cycle
# 0.31 stalled cycles per insn ( +- 34.07% ) [50.96%]
53,147,072 branches # 538.201 M/sec ( +- 31.19% ) [50.18%]
2,451,318 branch-misses # 4.61% of all branches ( +- 11.32% ) [49.95%]
141.101988701 seconds time elapsed ( +- 99.91% )
The patch has 3 parts:
An addition to TEST:_cmd() which wraps the test invocation inside a
"perf stat ..." as shown above, using a %cmdwrap template.
Option-parsing code, which handles several option forms:
1) -lxperf
writes file "$tstamp.perf" into ./t (after `cd $t && ...`)
2) -lxperf=4
adds "--repeat 4" to perf stat invocation.
3) -lxperf=(key=val) pairs
known key=value pairs (with =defaults given)
- tool=perf # path to tool
- cmd=stat # todo try others, perf record, etc
- reps=1 #
- rptdir # write reports to a separate data-repo, safe from git clean -dxf
- name # to convey branch-name and/or config-info, tstamp is prefixed,
- opts # passthru options to perf-stat
$ENV{TEST_OPTS} is split on /;/ and prepended to @ARGV before option
handling, so the following export will affect both make jobs.
export TEST_OPTS="-v;-lxperf=reps=3:rptdir=$HOME/perl/perfdata"
make test
make test_harness
NOTES:
Test script output under perf stat --repeat 4 violates the "plan just
once" rule of TAPv13 spec. It could be extended:
"The plan cannot appear in the middle of the output, only at
beginning or end. Multiple sets of (plan + tests) are allowed, if
all plans agree"
perf stat --repeat N causes perf to compute std-deviation(s) for each
measure. 1st example below is w/o repeat, 2nd with it, corresponding
to the TEST and harness examples above. I dont know how to read into
the numbers, but theres value in having them; they provide some
estimate of "noise-floor" on performance for a given test.
The -opts option passthru to perf stat allows "-x," to cause perf stat
to emit CSV output. This is easier to parse than the human readable
output, but doesnt include the computed/normalized values, nor the
name that the "Performance counter stats" (see examples) are for.
TODO
Probable rework of -lxperf option parsing
- no way now to get output to STDOUT, only to file. maybe not worth having
- keep (k=v:)+ form
- use name="%B-%G-%T.perf" by default "$branch-$sha1st6-$timestamp.perf"
- use wr=1 to save file
- dir=$HOME/perfruns (shorten rptdir)
Using reps=2 yields "FAILED--seen duplicate leader" on every test.
Fixing this requires changes to TEST's <results> processing loop. The
reps handling sub-option code is in this patch because it serves as a
test-case for further changes to optional behavior.
The command wrapper template is pretty minimal and constrained; the
output file name is "$dir/$timestamp$name.perfstat", chosen to match
with $timestamp.ttimes so that both file-types are browsable as a pair.
TEST_OPTS is split on ';', -lxperf is split on ':' into k=v pairs.
There may be scenarios where these delimiters are inconvenient, or the
simplistic parse is brittle.
Performance counter stats for './perl base/cond.t' (3 runs):
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
As explained in commit c537bcda1, $do_nothing's purpose is to prevent
TEST from actually running the tests, so that it can be required by
harness, and its routines reused.
This patch moves TEST's $do_nothing check after its option-handling
code, allowing those options to be used to alter the behavior of the
TEST subroutines when called by harness, the same as when they're
called from TEST::main.
TODO: With TEST's option handling done even for $do_nothing cases, it
consumes all options before harness sees them, so harness options -v
and -torture are broken. There are multiple ways to handle it, all
needing some consensus:
- harness does selective option scanning before requiring TEST. It
could either consume its options or pass them thru once it handles
them.
- both -v and -torture are implemented by TEST and harness, and mean
qualitatively the same thing. OTOH, any consumers of the respective
outputs may care about differences. Simply, could rename one of
each option to avoid the conflict.
- use of $TEST::verbose (or are namespaces also to be avoided?)
|
| |
|
|
| |
Add comments to var declarations, to help clarify loop operation.
|
| |
|
|
|
| |
Commit 0978c4bed computed a timestamp string as part of an output
filename, make it available for others.
|
| |
|
|
|
|
| |
$failure handling in <$results> processing loop does not need extra
subtlety of defined-ness checks - just give it an initial (false)
value, and check for truth.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
commit c96083ea8213 added our @found inside sub _find_tests,
but testing with strict exposes probs:
$ perl -cw -Ilib -Mstrict t/TEST
Variable "@found" is not imported at t/TEST line 198.
Variable "@found" is not imported at t/TEST line 202.
Global symbol "@found" requires explicit package name at t/TEST line 198.
Global symbol "@found" requires explicit package name at t/TEST line 202.
No point in arguing, hoist decl out to file scope.
|
| |
|
|
|
|
|
| |
Adding -w should be ok, despite peculiar code style; it avoids
require, unlike "use strict". Added risk of perl brokenness seems
vanishingly small. Fix one used-once warning thus exposed, and one
test in FileHandle.t surprised by t/TEST's new '-w' in the shebang.
|
| |
|
|
|
| |
This has never worked, as it would look for t/benchmark/*.t files in
the wrong place.
|
| |
|
|
|
|
| |
Add a new directory designed to hold performance / optimising tests
and infrastructure, and add the first test file, opcount.t, that
checks that a sub has the right numbers of particular op types
|
| |
|
|
| |
The Perl Toolchain Gang has agreed to maintain this.
|
| |
|
|
| |
Maintained by the Perl Toolchain Gang now
|
| |
|
|
| |
Maintained by the Perl Toolchain Gang now
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This will allow us to set up a deparse smoker to keep track of B::Deparse's
accuracy.
With this, the test run:
TEST_ARGS=-deparse make test
will "pass" currently.
If deparsing capabilities get worse, new failures will show up and should
be fixed or added to Porting/deparse-skips.txt.
If things get fixed, the test will fail noting that something has passed
when it shouldn't have, and the test in question should be removed from
Porting/deparse-skips.txt.
Use the KEEP_DEPARSE_FILES=1 environment setting to have the deparse
test files left around after a test run for examination/re-running.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Output for each test will be printed inline when it finishes.
Sample usage (loud):
TEST_JOBS=9 make test.valgrind
Sample usage (quiet):
VG_OPTS='-q --leak-check=no --show-reachable=no' TEST_JOBS=9 make test.valgrind
|
| |
|
|
|
|
|
|
| |
This reverts commit f0aff2daa44923f600f6e1f2429a2add2214a9d5
"Add support for test.valgrind parallel testing"
This patch was accidentally pushed (by me); it breaks some things. We
will wait to add this support until after 5.20
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
# New Ticket Created by Matthew Horsfall
# Please include the string: [perl #121431]
# in the subject line of all future correspondence about this issue.
# <URL: https://rt.perl.org/Ticket/Display.html?id=121431 >
This is a bug report for perl from wolfsage@gmail.com,
generated with the help of perlbug 1.39 running under perl 5.14.2.
-----------------------------------------------------------------
[Please describe your issue here]
The included patch allows test.valgrind to run tests in parallel.
Valgrind output for each test will be printed out after the test
completes, with the name of the test prefixing every line.
Example usage might be:
TEST_JOBS=8 make test.valgrind VALGRIND='valgrind -q' 2>&1 | tee out.txt
-q is needed to ensure only *errors* are captured, otherwise the output will
be much louder than it already is. (Perhaps this should be the default mode?)
[Please do not change anything below this line]
-----------------------------------------------------------------
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Remove the targets:
perl.pixie perl.pixie.atom perl.pixie.config perl.pixie.irix perl.third
perl.third.config
It's still possible to run the actions these targets "by hand", if desired.
This commit removes the convenience targets from the Makefile, reducing its
complexity. It also removes the related support scripts testall.atom and
thirdclean from Porting/
pixie is a performance analysis tool for Irix and Tru64
Third Degree is a memory checker tool for Tru64
Given that Tru64 went out of support at the end of 2012, and Irix goes out
of support at the end of 2013, it's very unlikely that anyone is still
actively profiling or debugging perl on either platform, and hence using
these targets. It's been several years since we've even had a regular bug
report from either platform.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In order to completely compile Perl, many modules must have been parsed
and compiled, so if there is a full perl, we know that things basically
work. The purpose of bailing out is that if these supposedly very base
level functionality tests don't work, there's no point in continuing.
But over the years, tests of more esoteric functionality have been
added here, and if one of them doesn't work, it still could be that Perl
pretty much does work.
I believe it would be best to move such non-basic tests elsewhere, but
that's work, and hasn't bitten us much so far; this change lessens the
severity of the biting even more. Where it will really bite is if
things are so bad that a full perl binary can't be compiled, and we are
trying to figure out why using minitest.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
Note:
Porting/core-cpan-diff refactored to use Archive::Tar
instead of Archive::Extract
|
| |
|
|
|
|
| |
directory t/opbasic.
For RT #115838
|
| | |
|
