diff options
Diffstat (limited to 'README.TESTING')
| -rw-r--r-- | README.TESTING | 69 |
1 files changed, 46 insertions, 23 deletions
diff --git a/README.TESTING b/README.TESTING index 2c2635e2e2..4fa6b4871f 100644 --- a/README.TESTING +++ b/README.TESTING @@ -14,10 +14,11 @@ Usage: make test "make test" basically executes "run-tests.php" script -under source root. Therefore you can execute the script -as follows +under the source root (parallel builds will not work). Therefore you +can execute the script as follows: -./sapi/cli/php -c /path/to/php.ini/ run-tests.php [ext/some_extension_name] +TEST_PHP_EXECUTABLE=sapi/cli/php \ +sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB] [Which "php" executable "make test" look for] @@ -37,38 +38,39 @@ executing "run-tests.php" script, you may get errors. [Which php.ini is used] ----------------------- - "make test" force to use php.ini-dist as default config file. If -you would like to test with other configuration file, user -"run-tests.php" script. - -Example: -./sapi/cli/php -c /path/to/php.ini/ run-tests.php ext/standard - -If you use php.ini other than php.ini-dist, you may see more failed -tests. + "make test" uses the same php.ini file as it would once installed. +The tests have been written to be independant of that php.ini file, +so if you find a test that is affected by a setting, please report +this, so we can address the issue. [Which test scripts are executed] --------------------------------- - "run-tests.php" ("make test") executes all test scripts by default -by looking all directory named "tests". If there are files have "phpt" -extension, "run-tests.php" takes test php code from the file and -executes it. + "run-tests.php" ("make test"), without any arguments executes all +test scripts by looking extracting all directories named "tests" +from the source root and any subdirectories below. If there are files, +which have a "phpt" extension, "run-tests.php" looks at the sections +in these files, determines whether it should run it, by evaluating +the 'SKIP' section. If the test is elligable for execution, the 'FILE' +section is extracted into a ".php" file, with a unique name and +executed. +When an argument is given or TESTS environment variable is set, the +GLOB is expanded by the shell and any file with extension "*.phpt" is +regarded as a testfile. Tester can easily execute tests selectively with as follows. Examples: -./sapi/cli/php -c /path/to/php.ini/ run-tests.php ext/mbstring -./sapi/cli/php -c /path/to/php.ini/ run-tests.php ext/mbstring/020.phpt +./sapi/cli/php run-tests.php ext/mbstring/* +./sapi/cli/php run-tests.php ext/mbstring/020.phpt [Test results] -------------- Test results are printed to standard output. If there is a failed test, -"run-tests.php" script saves the result, expected result and code -executed to the test script directory. For example, if -ext/myext/tests/myext.phpt is failed to pass, following files are -created: +the "run-tests.php" script saves the result, the expected result and the +code executed to the test script directory. For example, if +ext/myext/tests/myext.phpt fails to pass, the following files are created: ext/myext/tests/myext.php - actual testfile executed ext/myext/tests/myext.log - log of test execution (L) @@ -76,7 +78,13 @@ ext/myext/tests/myext.exp - expected output (E) ext/myext/tests/myext.out - output from test script (O) ext/myext/tests/myext.diff - diff of .out and .exp (D) - Tester can verify these files, if failed test is actually a bug or not. + Failed tests are always bugs. Either the test is bugged or not considering +factors applying to the tester's environment, or there is a bug in PHP. +If this is a known bug, we strive to provide bugnumbers, in either the +test name or the file name. You can check the status of such a bug, by +going to: http://bugs.php.net/12345 where 12345 is the bugnumber. +For clarity and automated processing, bugnumbers are prefixed by a hash +sign '#' in testnames and/or testcases are named bug12345.phpt. NOTE: The files generated by tests can be selected by setting the environment variable TEST_PHP_LOG_FORMAT. For each file you want to be @@ -86,6 +94,21 @@ The php file will be generated always. NOTE: You can set environment variable TEST_PHP_DETAILED to enable detailed test information. +[Automated testing] + If you like to keep up to speed, with latest developments and quality +assurance, setting the environment variable NO_INTERACTION to 1, will not +prompt the tester for any user input. + +Normally, the exit status of "make test" is zero, regardless of the results +of independant tests. Set the environment variable REPORT_EXIT_STATUS to 1, +to make "make test' set the exit statis ("$?") to non-zero, when a test has +failed. + +NOTE: the exit status of run-tests.php will be 1 when +REPORT_EXIT_STATUS is set. The result of "make test" may be higher +than that. At present, gmake 3.79.1 returns 2, so it is +advised to test for non-zero, rather then a specific value. + [Creating new test files] ------------------------- |