path: root/README.TESTING

[IMPORTANT NOTICE]
------------------
 Do _not_ ask to developers why some or all tests are failed under
your environment! Let us know if you find why it fails. Thank you.


[Testing Basics] 
----------------
 To execute test scripts, you must build PHP with some SAPI, then you
type "make test" to execute all or some test scripts saved under
"tests" directory under source root directory.

Usage:
make test

 "make test" basically executes "run-tests.php" script
under source root. Therefore you can execute the script
as follows

./sapi/cli/php -c /path/to/php.ini/ run-tests.php [ext/some_extension_name]


[Which "php" executable "make test" look for]
---------------------------------------------
 You must use TEST_PHP_EXECUTABLE environment variable to explicitly
select the php executable to be used to run the tests. That can either
be the CLI or CGI executable.

 "make test" executes "run-tests.php" script with "php" binary.  Some
test scripts such as session must be executed by CGI SAPI. Therefore,
you must build PHP with CGI SAPI to perform all tests.

NOTE: PHP binary executing "run-tests.php" and php binary used for
executing test scripts may differ. If you use different PHP binary for
executing "run-tests.php" script, you may get errors.


[Which php.ini is used]
-----------------------
 "make test" force to use php.ini-dist as default config file. If 
you would like to test with other configuration file, user 
"run-tests.php" script.

Example:
./sapi/cli/php -c /path/to/php.ini/ run-tests.php ext/standard

If you use php.ini other than php.ini-dist, you may see more failed
tests.


[Which test scripts are executed]
---------------------------------
 "run-tests.php" ("make test") executes all test scripts by default
by looking all directory named "tests". If there are files have "phpt"
extension, "run-tests.php" takes test php code from the file and 
executes it.

 Tester can easily execute tests selectively with as follows.

Examples:
./sapi/cli/php -c /path/to/php.ini/ run-tests.php ext/mbstring
./sapi/cli/php -c /path/to/php.ini/ run-tests.php ext/mbstring/020.phpt


[Test results]
--------------
 Test results are printed to standard output. If there is a failed test, 
"run-tests.php" script saves the result, expected result and code
executed to the test script directory. For example, if 
ext/myext/tests/myext.phpt is failed to pass, following files are 
created:

ext/myext/tests/myext.php   - actual testfile executed
ext/myext/tests/myext.log   - log of test execution (L)
ext/myext/tests/myext.exp   - expected output (E)
ext/myext/tests/myext.out   - output from test script (O)
ext/myext/tests/myext.diff  - diff of .out and .exp (D)

 Tester can verify these files, if failed test is actually a bug or not. 

NOTE: The files generated by tests can be selected by setting the
environment variable TEST_PHP_LOG_FORMAT. For each file you want to be
generated use the character in brackets as shown above (default is LEOD).
The php file will be generated always.

NOTE: You can set environment variable TEST_PHP_DETAILED to enable
detailed test information.


[Creating new test files]
-------------------------
 Writing test file is very easy if you are used to PHP. Test file
has following format. Here is a actual test file from iconv module.

===== ext/iconv/002.phpt =======
--TEST--
UCS4BE to ASCII
--SKIPIF--
<?php include('skipif.inc'); ?>
--FILE--
<?php include('002.inc'); ?>
--EXPECT--
&#97;&#98;&#99;&#100;
abcd
===== end of ext/iconv/002.phpt =======

 As you can see the file has the following sections:

"--TEST--"    is title of the test.
"--SKIPIF--"  is condition when to skip this test (optional).
"--POST--"    is POST variable passed to test script (optional).
"--GET--"     is GET variable passed to test script (optional).
"--INI--"     each line contains an ini setting e.g. foo=bar (optional).
"--FILE--"    is the test script.
"--EXPECT--"  is the expected output from the test script.
"--EXPECTF--" this is the alternate form of --EXPECT--. The difference 
              is is that this form uses sscanf for output validation.

ext/iconv/002.phpt uses 2 files. "skipif.inc" is used to skip
test when test cannot be performed such as iconv module is not
compiled or loaded. 

==== ext/iconv/skipif.inc ====
<?php
// This script prints "skip" if condition does not meet.

if (!extension_loaded("iconv") && ini_get("enable_dl")) {
  $dlext = (substr(PHP_OS, 0, 3) == "WIN") ? ".dll" : ".so";
  @dl("iconv$dlext");
}
if (!extension_loaded("iconv")) {
    die("skip\n");
}
?>
==== end of ext/iconv/skipif.inc ====

ext/inconv/002.inc is the test script. "run-tests.php" script
executes test script with CGI executable.

==== ext/iconv/002.inc ===
<?php
/*
Expected output:
&#97;&#98;&#99;&#100;
abcd
*/

   $s = unpack("V*", iconv("ascii","UCS-4LE", "abcd"));
   foreach($s as $c) { print "&#$c;"; } print "\n";

   $s = pack("NNNN", 97, 98, 99, 100);
   $q = iconv("UCS-4BE", "ascii", $s);
   print $q; print "\n";
?>
=== end of ext/iconv/002.inc ===

Test script and SKIPIF code should be directly written into *.phpt.
However, it is recommended to use include files when more test scripts
depend on the same SKIPIF code or when certain test files need the same
values for some input.


/ext/standard/tests/strings 003.phpt is a good example for using EXPECTF 
instead of EXPECT. From time to time the algorythmn used for shuffle 
changed and sometimes the machine used to execute the code has influence 
on the result of shuffle. But it always retuns a three character string 
detectable by %s. Other forms scanable are %i for integers, %f for 
floating point values and %x for hexadecimal values.

==== /ext/standard/tests/strings 003.phpt ===
--TEST--
Testing str_shuffle.
--FILE--
<?php
$s = '123';
var_dump(str_shuffle($s));
var_dump($s);
?>
--EXPECTF--
string(3) %s
string(3) "123"
==== end of /ext/standard/tests/strings 003.phpt ===

[How to help us]
----------------
 If you find bug in PHP, you can submit bug report AND test script 
for us. You don't have to write complete script, just give us test
script with following format. Please test the script and make sure
you write the correct ACTUAL OUTPUT and EXPECTED OUTPUT before you
submit.

<?php
/* 
Bug #12345
substr() bug. Do not return expected string.

ACTUAL OUTPUT
XYXA

EXPECTED OUTPUT
ABCD
*/

$str = "XYZABCD";
echo substr($str,3,7);

?>