summaryrefslogtreecommitdiff
path: root/run-tests.php
Commit message (Collapse)AuthorAgeFilesLines
* run-tests: use the EXTENSIONS section for skippingMax Semenik2021-03-221-2/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, most skip checks are just for making sure an extension is available. Even with recent addition of skip caching, this makes tests needlessly slow: * Checks for the same extension in its tests can have small differences impacting cacheability. * Even identical skip checks in two tests can still be executed twice if they're run by different workers. To remedy this, I'm repurposing the existing --EXTENSIONS-- section of .phpt files to specify wjich extensions are required for current test to run. Current behavior: 1) If the extension is already visible to PHP, all is good 2) If it isn't, assume it's present as a shared module and attempt to add it via a command line parameter 3) If that works, all is good 4) If it doesn't, PHP fails with a cryptic error message trying to execute the test itself After this commit: 1) and 2) are kept unchanged 3) Check if shared extension file from 2) is actually present 4) Skip the test if it isn't Other benefits include clear skip reasons (vs. sometimes none in many current skip checks) and moving test information from code to metadata, opening more opportunities for search and analysis. Since --EXTENSIONS-- is barely ever used, this change poses no risk of hidden failures. As a demonstration of the new approach, this commit migrates one extension to it. If merged, I will migrate other extensions in subsequent PRs. Closes GH-6787.
* run-tests.php: class for test file loadingMax Semenik2021-03-161-251/+345
| | | | | | | | | | | | | | | | | This moves a bunch of code outside of run_tests(), making it a bit more manageable. Additionally, accessors provide better readability than isset() and friends. This is a minimal patch that moves the code but does not refactor much. For the sake of reviewing experience, it does not involve further refactoring which could include: * Removing setSection() * Fixing up the mess with hasSection() vs. sectionNotEmpty(), only one of which is really needed. * Moving more repetitive code into the new class. All of this will be done with later commits. Closes GH-6678.
* Make the nocache tag actually workNikita Popov2021-03-041-1/+3
| | | | | | | | If nocache is used, then we should return an empty string result, so the test doesn't get marked as borked. Also limit to an occurrence at the start, just like other skip tags.
* Add mechanism to skip the skipif cacheNikita Popov2021-03-041-1/+3
| | | | | | If the SKIPIF section return nocache, don't cache the result. This is used by tests that perform complex test set in conjunction with skip checks, and need to be evaluated every time.
* run-tests.php: Preserve .php on leak in one more placeNikita Popov2021-03-041-1/+1
|
* run-tests: drop support for ancient Valgrind versionsMax Semenik2021-03-011-7/+1
| | | | | | | | Valgrind 3.3.0 was released in 2007, not even RHEL has crap older than this. It could be argued that 3.8.0, released in 2012, could be a safe cutoff too. Closes GH-6728.
* Merge branch 'PHP-8.0'Christoph M. Becker2021-02-251-0/+3
|\ | | | | | | | | * PHP-8.0: Capture Microsoft-defined HRESULT exit codes exit codes
| * Merge branch 'PHP-7.4' into PHP-8.0Christoph M. Becker2021-02-251-0/+3
| |\ | | | | | | | | | | | | * PHP-7.4: Capture Microsoft-defined HRESULT exit codes exit codes
| | * Capture Microsoft-defined HRESULT exit codes exit codesDylan K. Taylor2021-02-251-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | The lack of such a check leads to false-passes of tests on Windows which expect no output, but produce a segfault or similar issue. I discovered this a while ago due to bad tests in an extension I maintain. Closes GH-6722.
* | | run-tests: add skip cacheMax Semenik2021-02-231-13/+82
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently every --SKIPIF-- section in every test file results in 1 extra execution of PHP, every --EXTENSIONS-- section - in 2 executions. This is quite wasteful, as skip checking code is extremely repetitive and extensions are fixed for every binary/ini/command parameters combination. This patch adds caching to all such checks. On my machine, the gains are quite noticeable: 36s instead of 43s with -j16, 292s instead of 337s without concurrency. Cache stats are 3780 hits, 1247 misses in the latter case. In the future, tests could be adjusted to have more uniform skip checks to improve performance even more. Closes GH-6681.
* | | Merge branch 'PHP-8.0'Christoph M. Becker2021-02-231-1/+1
|\ \ \ | |/ / | | | | | | | | | * PHP-8.0: run-tests: fixed exit code not being set on BORKED tests
| * | Merge branch 'PHP-7.4' into PHP-8.0Christoph M. Becker2021-02-231-1/+1
| |\ \ | | |/ | | | | | | | | | * PHP-7.4: run-tests: fixed exit code not being set on BORKED tests
| | * Merge branch 'PHP-7.3' into PHP-7.4Christoph M. Becker2021-02-231-1/+1
| | |\ | | | | | | | | | | | | | | | | * PHP-7.3: run-tests: fixed exit code not being set on BORKED tests
| | | * run-tests: fixed exit code not being set on BORKED testsPHP-7.3Dylan K. Taylor2021-02-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | When no test paths are specified this shows up when 'make test' is used on a PECL extension without specifying tests to run (or in php-src too, I guess...) Closes GH-6717.
| | | * Put Xdebug 3 into 'off' mode for run-tests.phpDerick Rethans2020-05-181-1/+1
| | | |
* | | | Merge branch 'PHP-8.0'Nikita Popov2021-02-171-0/+1
|\ \ \ \ | |/ / / | | | | | | | | | | | | * PHP-8.0: Color LEAK&FAIL like a failure
| * | | Color LEAK&FAIL like a failureNikita Popov2021-02-171-0/+1
| | | |
* | | | Remove stray mentions of mbstring.func_overloadMax Semenik2021-02-151-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | This feature has been completely removed. Closes GH-6688.
* | | | Deprecate passing null to non-nullable arg of internal functionNikita Popov2021-02-111-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This deprecates passing null to non-nullable scale arguments of internal functions, with the eventual goal of making the behavior consistent with userland functions, where null is never accepted for non-nullable arguments. This change is expected to cause quite a lot of fallout. In most cases, calling code should be adjusted to avoid passing null. In some cases, PHP should be adjusted to make some function arguments nullable. I have already fixed a number of functions before landing this, but feel free to file a bug if you encounter a function that doesn't accept null, but probably should. (The rule of thumb for this to be applicable is that the function must have special behavior for 0 or "", which is distinct from the natural behavior of the parameter.) RFC: https://wiki.php.net/rfc/deprecate_null_to_scalar_internal_arg Closes GH-6475.
* | | | run-tests: fix JUnit countsMax Semenik2021-02-111-1/+7
| | | | | | | | | | | | | | | | | | | | In 19680f886f, I forgot about my own TODO. At least, now I understand what this line was about :P
* | | | run-tests.php: move JUnit stuff into a classMax Semenik2021-02-081-263/+260
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This is part one of my work that was announced at https://externals.io/message/110391 Closes GH-6671.
* | | | Fix environment variable not work in run-tests.phptwosee2021-02-011-2/+2
| | | | | | | | | | | | | | | | Closes GH-6657.
* | | | Create .php and .sh on valgrind failureNikita Popov2021-02-011-17/+20
| | | |
* | | | Show slow SKIPIF sections as wellNikita Popov2020-12-211-1/+13
| | | |
* | | | Determine run-tests executables consistentlyNikita Popov2020-12-011-78/+44
| | | |
* | | | Don't pass null to strlen()Nikita Popov2020-12-011-1/+1
| | | |
* | | | Merge branch 'PHP-8.0'Nikita Popov2020-11-301-4/+6
|\ \ \ \ | |/ / / | | | | | | | | | | | | * PHP-8.0: Fixed bug #80437
| * | | Fixed bug #80437Nikita Popov2020-11-301-4/+6
| | | | | | | | | | | | | | | | Only chmod if the file has been created.
* | | | Add --repeat flag to usage info of run-tests.phpChristoph M. Becker2020-11-151-0/+4
| | | | | | | | | | | | | | | | Closes GH-6429.
* | | | Add --repeat testing modeNikita Popov2020-10-301-16/+69
|/ / / | | | | | | | | | | | | | | | | | | | | | | | | This testing mode executes the test multiple times in the same process (but in different requests). It is primarily intended to catch tracing JIT bugs, but also catches state leaks across requests. Closes GH-6365.
* | | Don't set xdebug.default_enable in run-testsbashkarev2020-10-261-1/+1
| | | | | | | | | | | | | | | | | | This throws a deprecation warning in XDebug 3. Closes GH-6324.
* | | Skip arginfo test under msanNikita Popov2020-10-151-0/+4
| | | | | | | | | | | | | | | Msan is missing interceptors for some functions that result in false positives.
* | | Remove unused --temp-urlbase option from run-tests.phpNikita Popov2020-09-231-8/+1
| | | | | | | | | | | | This was part of the removed HTML functionality.
* | | Fix: Gracefully handle empty user input in run-tests.phpBerbe2020-08-241-1/+1
| | | | | | | | | | | | Closes GH-6016.
* | | Add `run-tests.php --context [n]` option.Tyson Andre2020-08-161-7/+68
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Mentioned in https://github.com/php/php-src/pull/5965#discussion_r467621123 This PR proposes 3 lines of context so the impact can be seen in tests. Other `diff` programs show around 3 lines of context. (This helps indicate exactly which position a test should be updated to add a new expected line at) Use the mapping for choosing order to display diffs Properly include context in cases where the expected output had more lines than the actual output, e.g. ``` --FILE-- A A1 A C NEARBY --EXPECTF-- A B A1 B A B A B NEARBY ``` Closes GH-5968
* | | Colorize --show-diff, only on stdoutTyson Andre2020-08-131-0/+6
| | | | | | | | | | | | | | | | | | Don't add colors to the saved `.diff` file. Related to a41cf3e1d2ed3ae38ad6115a8b65e5f14dae0b49
* | | Support NO_COLOR environment variable in run-tests.phpTyson Andre2020-08-121-1/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | And add a --color option for run-tests.php See https://no-color.org/ > an informal standard is hereby proposed: > > All command-line software which outputs text with ANSI color added should check > for the presence of a `NO_COLOR` environment variable that, when present > (regardless of its value), prevents the addition of ANSI color.
* | | Revert "Make diff section contents red(-)/green(+) in run-tests.php"Nikita Popov2020-08-101-26/+6
| | | | | | | | | | | | | | | | | | | | | This reverts commit 58489bd3b93e10c150933eefac05297b48213eb3. This also uses ANSI escape code coloring for .diff files, where it is not meaningful.
* | | Make diff section contents red(-)/green(+) in run-tests.phpTyson Andre2020-08-091-6/+26
| | | | | | | | | | | | | | | | | | | | | | | | This uses green for lines with `+` and red for lines with `-`. Colors(Red and Green) would make the failure causes stand out visually when scrolling through errors. Closes GH-5965
* | | Use short array values in run-tests.phpTyson Andre2020-08-091-88/+88
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The run-tests.php file already requires php 7 syntax. Done automatically with phpcbf with no other changes done/planned. This file is already using short arrays in other places. run-tests has already been substantially refactored in PHP 8.0 compared to PHP 7.4. This makes merge conflicts more likely, but changes to run-tests.php on older versions are less frequent bug fixes.
* | | Add colours to test results in run-tests.phpGeorge Peter Banyard2020-08-071-3/+29
| | | | | | | | | | | | | | | | | | | | | | | | The test runner will attempt to colourise the result of each test The --no-color option is introduced to disable this feature. Closes GH-5901
* | | Add type information to test-runnerGeorge Peter Banyard2020-08-061-72/+114
| | | | | | | | | | | | Closes GH-5942
* | | [RFC] Make string length for getTraceAsString() configurableTyson Andre2020-07-251-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add a `zend.exception_string_param_max_len` ini setting. (same suffix as `log_errors_max_len`) Allow values between 0 and 1000000 bytes. For example, with zend.exception_string_param_max_len=0, "" would represent the empty string, and "..." would represent something longer than the empty string. Previously, this was hardcoded as exactly 15 bytes. Discussion: https://externals.io/message/110717 Closes GH-5769
* | | Use small counter values for testingDmitry Stogov2020-07-081-0/+4
| | |
* | | Removing HTML Functionality from run-tests.phpPaul Dragoonis2020-06-191-119/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | As discussed on GH-5632, the HTML functionality does not appear to be in active use. For HTML rendering of test results, it is suggested to instead use the JUnit integration, in combination with your favorite JUnit viewer. Closes GH-5705.
* | | Optimize parallel test runner startupNikita Popov2020-06-161-13/+14
| | | | | | | | | | | | | | | | | | | | | Don't connect to each worker individually: First spawn all processes and then establish connections in any order. This avoids a ~1s wait when running on many cores.
* | | run-tests: refactorGerard Roche2020-05-221-29/+48
| | |
* | | run-tests: echo message and exit if no tests foundGerard Roche2020-05-221-0/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The test runner currently defaults to running the entire test suite if no selected tests can be found. This can be unexpected. For example the ext/mysqlnd/ directory has no tests, if you specify that directory when testing the entire test suite will be run. run-tests.php [options] ext/mysqlnd/ Closes GH-5605.
* | | Add --file-cache-prime/use options to run-testsNikita Popov2020-05-201-5/+38
| | | | | | | | | | | | | | | | | | | | | | | | --file-cache-prime populates the file cache, --file-cache-use uses the file cache. And fix a number of tests to run under file cache or disabled timestamp validation.
* | | Slightly deduplicate run-tests codeNikita Popov2020-05-201-66/+54
| | |
View Lorry Controller Status