[IMPORTANT NOTICE] ------------------ Do _not_ ask to developers why some or all tests are failed under your environment! Let us know if you find why it fails. Thank you. [Testing Basics] ---------------- To execute test scripts, you must build PHP with some SAPI, then you type "make test" to execute all or some test scripts saved under "tests" directory under source root directory. Usage: make test "make test" basically executes "run-tests.php" script under the source root (parallel builds will not work). Therefore you can execute the script as follows: TEST_PHP_EXECUTABLE=sapi/cli/php \ sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB] [Which "php" executable "make test" look for] --------------------------------------------- You must use TEST_PHP_EXECUTABLE environment variable to explicitly select the php executable to be used to run the tests. That can either be the CLI or CGI executable. "make test" executes "run-tests.php" script with "php" binary. Some test scripts such as session must be executed by CGI SAPI. Therefore, you must build PHP with CGI SAPI to perform all tests. NOTE: PHP binary executing "run-tests.php" and php binary used for executing test scripts may differ. If you use different PHP binary for executing "run-tests.php" script, you may get errors. [Which php.ini is used] ----------------------- "make test" uses the same php.ini file as it would once installed. The tests have been written to be independent of that php.ini file, so if you find a test that is affected by a setting, please report this, so we can address the issue. [Which test scripts are executed] --------------------------------- "run-tests.php" ("make test"), without any arguments executes all test scripts by extracting all directories named "tests" from the source root and any subdirectories below. If there are files, which have a "phpt" extension, "run-tests.php" looks at the sections in these files, determines whether it should run it, by evaluating the 'SKIP' section. If the test is eligible for execution, the 'FILE' section is extracted into a ".php" file (with the same name besides the extension) and gets executed. When an argument is given or TESTS environment variable is set, the GLOB is expanded by the shell and any file with extension "*.phpt" is regarded as a test file. Tester can easily execute tests selectively with as follows. Examples: ./sapi/cli/php run-tests.php ext/mbstring/* ./sapi/cli/php run-tests.php ext/mbstring/020.phpt [Test results] -------------- Test results are printed to standard output. If there is a failed test, the "run-tests.php" script saves the result, the expected result and the code executed to the test script directory. For example, if ext/myext/tests/myext.phpt fails to pass, the following files are created: ext/myext/tests/myext.php - actual test file executed ext/myext/tests/myext.log - log of test execution (L) ext/myext/tests/myext.exp - expected output (E) ext/myext/tests/myext.out - output from test script (O) ext/myext/tests/myext.diff - diff of .out and .exp (D) Failed tests are always bugs. Either the test is bugged or not considering factors applying to the tester's environment, or there is a bug in PHP. If this is a known bug, we strive to provide bug numbers, in either the test name or the file name. You can check the status of such a bug, by going to: http://bugs.php.net/12345 where 12345 is the bug number. For clarity and automated processing, bug numbers are prefixed by a hash sign '#' in test names and/or test cases are named bug12345.phpt. NOTE: The files generated by tests can be selected by setting the environment variable TEST_PHP_LOG_FORMAT. For each file you want to be generated use the character in brackets as shown above (default is LEOD). The php file will be generated always. NOTE: You can set environment variable TEST_PHP_DETAILED to enable detailed test information. [Automated testing] If you like to keep up to speed, with latest developments and quality assurance, setting the environment variable NO_INTERACTION to 1, will not prompt the tester for any user input. Normally, the exit status of "make test" is zero, regardless of the results of independent tests. Set the environment variable REPORT_EXIT_STATUS to 1, and "make test" will set the exit status ("$?") to non-zero, when an individual test has failed. Example script to be run by cron(1): ========== qa-test.sh ============= #!/bin/sh CO_DIR=$HOME/cvs/php4 MYMAIL=qa-test@domain.com TMPDIR=/var/tmp TODAY=`date +"%Y%m%d"` # Make sure compilation enviroment is correct CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre' export MAKE=gmake export CC=gcc # Set test environment export NO_INTERACTIVITY=1 export REPORT_EXIT_STATUS=1 cd $CO_DIR cvs update . >>$TMPDIR/phpqatest.$TODAY ./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE $MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1 if test $? -gt 0 then cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL fi ========== end of qa-test.sh ============= NOTE: the exit status of run-tests.php will be 1 when REPORT_EXIT_STATUS is set. The result of "make test" may be higher than that. At present, gmake 3.79.1 returns 2, so it is advised to test for non-zero, rather then a specific value. [Creating new test files] ------------------------- Writing test file is very easy if you are used to PHP. Here is an actual test file from standard module. ===== ext/standard/tests/strings/strtr.phpt ======= --TEST-- strtr() function --FILE-- "hi", "hi"=>"hello", "a"=>"A", "world"=>"planet"); var_dump(strtr("# hi all, I said hello world! #", $trans)); ?> --EXPECT-- string(32) "# hello All, I sAid hi planet! #" ===== ext/standard/tests/strings/strtr.phpt ======= As you can see the file is devided into several sections. Below is a list of all possible sections: "--TEST--" is title of the test (required). "--SKIPIF--" is condition when to skip this test (optional). "--POST--" is POST variable passed to test script (optional). "--GET--" is GET variable passed to test script (optional). "--INI--" each line contains an ini setting e.g. foo=bar (optional). "--FILE--" is the test script (required). "--EXPECT--" is the expected output from the test script (required). "--EXPECTF--" is an alternative of --EXPECT--. The difference is that this form uses sscanf for output validation (alternative). "--EXPECTREGEX--" is an alternative of --EXPECT--. This form allows the tester to specify the result in a regular expression (alternative). A test must at least contain the sections TEST, FILE and either EXPECT or EXPECTF. When a test is called run-test.php takes the name from the TEST section and writes the FILE section into a ".php" file with the same name as the ".phpt" file. This ".php" file will then be executed and its output compared to the contents of the EXPECT section. It is a good idea to generate output with var_dump() calls. /ext/standard/tests/strings/str_shuffle.phpt is a good example for using EXPECTF instead of EXPECT. From time to time the algorithm used for shuffle changed and sometimes the machine used to execute the code has influence on the result of shuffle. But it always returns a three character string detectable by %s. Other scan-able forms are %i for integers, %d for numbers only, %f for floating point values, %c for single characters and %x for hexadecimal values. ==== /ext/standard/tests/strings/str_shuffle.phpt === --TEST-- Testing str_shuffle. --FILE-- --EXPECTF-- string(3) %s string(3) "123" ==== end of /ext/standard/tests/strings/str_shuffle.phpt === /ext/standard/tests/strings/strings001.phpt is a good example for using EXPECTREGEX instead of EXPECT. This test also shows that in EXPECTREGEX some characters need to be escaped since otherwise they would be interpreted as a regular expression. ==== /ext/standard/tests/strings/strings001.phpt === --TEST-- Test whether strstr() and strrchr() are binary safe. --FILE-- --EXPECTREGEX-- string\(18\) \"nica\x00turska panica\" string\(19\) \" nica\x00turska panica\" ==== end of /ext/standard/tests/strings/strings001.phpt === Some tests depend on modules or functions available only in certain versions or they even require minimum version of php or zend. These tests should be skipped when the requirement cannot be fullfilled. To achieve this you can use the SKIPIF section. To tell run-test.php that your test should be skipped the SKIPIF section must print out the word "skip" followed by a reason why the test should skip. ==== /ext/exif/tests/exif005.phpt === --TEST-- Check for exif_read_data, unusual IFD start --SKIPIF-- --FILE-- --EXPECT-- array(2) { ["ImageDescription"]=> string(11) "Ifd00000009" ["DateTime"]=> string(19) "2002:10:18 20:06:00" } ==== end of /ext/exif/tests/exif005.phpt === Test script and SKIPIF code should be directly written into *.phpt. However, it is recommended to use include files when more test scripts depend on the same SKIPIF code or when certain test files need the same values for some input. But no file used by any test should have one of the following extensions: ".php", ".log", ".exp", ".out" or ".diff". Tests should be named according to the following list: Class: Name: Example: Tests for bugs bug.phpt bug17123.phpt Tests for functions .phpt dba_open.phpt General tests for extensions .phpt dba3.phpt When you use an include file for the SKIPIF section it should be named "skipif.inc" and an include file used in the FILE section of many tests should be named "test.inc". NOTE: All tests should run correctly with error_reporting(E_ALL) and display_errors=1. This is the default when called from run-test.php. If you have a good reason for lowering the error reporting, use --INI-- section and comment this in your testcode. NOTE: If your test intentionally generates a PHP warning message use $php_errormsg variable, which you can then output. This will result in a consistent error message output across all platforms and PHP configurations, preventing your test from failing due inconsistencies in the error message content. [How to help us] ---------------- If you find bug in PHP, you can submit bug report AND test script for us. You don't have to write complete script, just give us test script with following format. Please test the script and make sure you write the correct ACTUAL OUTPUT and EXPECTED OUTPUT before you submit.