diff options
Diffstat (limited to 'README.TESTING')
| -rw-r--r-- | README.TESTING | 307 |
1 files changed, 0 insertions, 307 deletions
diff --git a/README.TESTING b/README.TESTING deleted file mode 100644 index ca21f3ae78..0000000000 --- a/README.TESTING +++ /dev/null @@ -1,307 +0,0 @@ -[IMPORTANT NOTICE] ------------------- - Do _not_ ask to developers why some or all tests are failed under -your environment! Let us know if you find why it fails. Thank you. - - -[Testing Basics] ----------------- - To execute test scripts, you must build PHP with some SAPI, then you -type "make test" to execute all or some test scripts saved under -"tests" directory under source root directory. - -Usage: -make test - - "make test" basically executes "run-tests.php" script -under the source root (parallel builds will not work). Therefore you -can execute the script as follows: - -TEST_PHP_EXECUTABLE=sapi/cli/php \ -sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB] - - -[Which "php" executable "make test" look for] ---------------------------------------------- - You must use TEST_PHP_EXECUTABLE environment variable to explicitly -select the php executable to be used to run the tests. That can either -be the CLI or CGI executable. - - "make test" executes "run-tests.php" script with "php" binary. Some -test scripts such as session must be executed by CGI SAPI. Therefore, -you must build PHP with CGI SAPI to perform all tests. - -NOTE: PHP binary executing "run-tests.php" and php binary used for -executing test scripts may differ. If you use different PHP binary for -executing "run-tests.php" script, you may get errors. - - -[Which php.ini is used] ------------------------ - "make test" uses the same php.ini file as it would once installed. -The tests have been written to be independent of that php.ini file, -so if you find a test that is affected by a setting, please report -this, so we can address the issue. - - -[Which test scripts are executed] ---------------------------------- - "run-tests.php" ("make test"), without any arguments executes all -test scripts by extracting all directories named "tests" -from the source root and any subdirectories below. If there are files, -which have a "phpt" extension, "run-tests.php" looks at the sections -in these files, determines whether it should run it, by evaluating -the 'SKIP' section. If the test is eligible for execution, the 'FILE' -section is extracted into a ".php" file (with the same name besides -the extension) and gets executed. -When an argument is given or TESTS environment variable is set, the -GLOB is expanded by the shell and any file with extension "*.phpt" is -regarded as a test file. - - Tester can easily execute tests selectively with as follows. - -Examples: -./sapi/cli/php run-tests.php ext/mbstring/* -./sapi/cli/php run-tests.php ext/mbstring/020.phpt - - -[Test results] --------------- - Test results are printed to standard output. If there is a failed test, -the "run-tests.php" script saves the result, the expected result and the -code executed to the test script directory. For example, if -ext/myext/tests/myext.phpt fails to pass, the following files are created: - -ext/myext/tests/myext.php - actual test file executed -ext/myext/tests/myext.log - log of test execution (L) -ext/myext/tests/myext.exp - expected output (E) -ext/myext/tests/myext.out - output from test script (O) -ext/myext/tests/myext.diff - diff of .out and .exp (D) - - Failed tests are always bugs. Either the test is bugged or not considering -factors applying to the tester's environment, or there is a bug in PHP. -If this is a known bug, we strive to provide bug numbers, in either the -test name or the file name. You can check the status of such a bug, by -going to: http://bugs.php.net/12345 where 12345 is the bug number. -For clarity and automated processing, bug numbers are prefixed by a hash -sign '#' in test names and/or test cases are named bug12345.phpt. - -NOTE: The files generated by tests can be selected by setting the -environment variable TEST_PHP_LOG_FORMAT. For each file you want to be -generated use the character in brackets as shown above (default is LEOD). -The php file will be generated always. - -NOTE: You can set environment variable TEST_PHP_DETAILED to enable -detailed test information. - -[Automated testing] - If you like to keep up to speed, with latest developments and quality -assurance, setting the environment variable NO_INTERACTION to 1, will not -prompt the tester for any user input. - -Normally, the exit status of "make test" is zero, regardless of the results -of independent tests. Set the environment variable REPORT_EXIT_STATUS to 1, -and "make test" will set the exit status ("$?") to non-zero, when an -individual test has failed. - -Example script to be run by cron(1): -========== qa-test.sh ============= -#!/bin/sh - -CO_DIR=$HOME/cvs/php4 -MYMAIL=qa-test@domain.com -TMPDIR=/var/tmp -TODAY=`date +"%Y%m%d"` - -# Make sure compilation enviroment is correct -CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre' -export MAKE=gmake -export CC=gcc - -# Set test environment -export NO_INTERACTIVITY=1 -export REPORT_EXIT_STATUS=1 - -cd $CO_DIR -cvs update . >>$TMPDIR/phpqatest.$TODAY -./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE -$MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1 -if test $? -gt 0 -then - cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL -fi -========== end of qa-test.sh ============= - -NOTE: the exit status of run-tests.php will be 1 when -REPORT_EXIT_STATUS is set. The result of "make test" may be higher -than that. At present, gmake 3.79.1 returns 2, so it is -advised to test for non-zero, rather then a specific value. - - -[Creating new test files] -------------------------- - Writing test file is very easy if you are used to PHP. - -Here is an actual test file from standard module. - -===== ext/standard/tests/strings/strtr.phpt ======= ---TEST-- -strtr() function ---FILE-- -<?php -/* Do not change this test it is a REATME.TESTING example. */ -$trans = array("hello"=>"hi", "hi"=>"hello", "a"=>"A", "world"=>"planet"); -var_dump(strtr("# hi all, I said hello world! #", $trans)); -?> ---EXPECT-- -string(32) "# hello All, I sAid hi planet! #" -===== ext/standard/tests/strings/strtr.phpt ======= - -As you can see the file is devided into several sections. Below is a -list of all possible sections: - -"--TEST--" is title of the test (required). -"--SKIPIF--" is condition when to skip this test (optional). -"--POST--" is POST variable passed to test script (optional). -"--GET--" is GET variable passed to test script (optional). -"--INI--" each line contains an ini setting e.g. foo=bar (optional). -"--FILE--" is the test script (required). -"--EXPECT--" is the expected output from the test script (required). -"--EXPECTF--" is an alternative of --EXPECT--. The difference is that - this form uses sscanf for output validation (alternative). -"--EXPECTREGEX--" is an alternative of --EXPECT--. This form allows the tester - to specify the result in a regular expression (alternative). - -A test must at least contain the sections TEST, FILE and either EXPECT -or EXPECTF. When a test is called run-test.php takes the name from the -TEST section and writes the FILE section into a ".php" file with the -same name as the ".phpt" file. This ".php" file will then be executed -and its output compared to the contents of the EXPECT section. It is a -good idea to generate output with var_dump() calls. - -/ext/standard/tests/strings/str_shuffle.phpt is a good example for using -EXPECTF instead of EXPECT. From time to time the algorithm used for shuffle -changed and sometimes the machine used to execute the code has influence -on the result of shuffle. But it always returns a three character string -detectable by %s. Other scan-able forms are %i for integers, %d for numbers -only, %f for floating point values, %c for single characters and %x for -hexadecimal values. - -==== /ext/standard/tests/strings/str_shuffle.phpt === ---TEST-- -Testing str_shuffle. ---FILE-- -<?php -/* Do not change this test it is a REATME.TESTING example. */ -$s = '123'; -var_dump(str_shuffle($s)); -var_dump($s); -?> ---EXPECTF-- -string(3) %s -string(3) "123" -==== end of /ext/standard/tests/strings/str_shuffle.phpt === - -/ext/standard/tests/strings/strings001.phpt is a good example for using -EXPECTREGEX instead of EXPECT. This test also shows that in EXPECTREGEX -some characters need to be escaped since otherwise they would be -interpreted as a regular expression. - -==== /ext/standard/tests/strings/strings001.phpt === ---TEST-- -Test whether strstr() and strrchr() are binary safe. ---FILE-- -<?php -/* Do not change this test it is a REATME.TESTING example. */ -$s = "alabala nica".chr(0)."turska panica"; -var_dump(strstr($s, "nic")); -var_dump(strrchr($s," nic")); -?> ---EXPECTREGEX-- -string\(18\) \"nica\x00turska panica\" -string\(19\) \" nica\x00turska panica\" -==== end of /ext/standard/tests/strings/strings001.phpt === - -Some tests depend on modules or functions available only in certain versions -or they even require minimum version of php or zend. These tests should be -skipped when the requirement cannot be fullfilled. To achieve this you can -use the SKIPIF section. To tell run-test.php that your test should be skipped -the SKIPIF section must print out the word "skip" followed by a reason why -the test should skip. - -==== /ext/exif/tests/exif005.phpt === ---TEST-- -Check for exif_read_data, unusual IFD start ---SKIPIF-- -<?php - if (!extension_loaded('exif')) print 'skip exif extension not available'; -?> ---FILE-- -<?php -/* Do not change this test it is a REATME.TESTING example. - * test5.jpg is a 1*1 image that contains an Exif section with ifd = 00000009h - */ -$image = exif_read_data('./ext/exif/tests/test5.jpg','',true,false); -var_dump($image['IFD0']); -?> ---EXPECT-- -array(2) { - ["ImageDescription"]=> - string(11) "Ifd00000009" - ["DateTime"]=> - string(19) "2002:10:18 20:06:00" -} -==== end of /ext/exif/tests/exif005.phpt === - -Test script and SKIPIF code should be directly written into *.phpt. However, -it is recommended to use include files when more test scripts depend on the -same SKIPIF code or when certain test files need the same values for some -input. But no file used by any test should have one of the following -extensions: ".php", ".log", ".exp", ".out" or ".diff". - -Tests should be named according to the following list: - -Class: Name: Example: -Tests for bugs bug<bugid>.phpt bug17123.phpt -Tests for functions <functionname>.phpt dba_open.phpt -General tests for extensions <extname><no>.phpt dba3.phpt - -When you use an include file for the SKIPIF section it should be named -"skipif.inc" and an include file used in the FILE section of many tests -should be named "test.inc". - -NOTE: All tests should run correctly with error_reporting(E_ALL) and -display_errors=1. This is the default when called from run-test.php. -If you have a good reason for lowering the error reporting, use --INI-- -section and comment this in your testcode. - -NOTE: If your test intentionally generates a PHP warning message use -$php_errormsg variable, which you can then output. This will result -in a consistent error message output across all platforms and PHP -configurations, preventing your test from failing due inconsistencies -in the error message content. - -[How to help us] ----------------- - If you find bug in PHP, you can submit bug report AND test script -for us. You don't have to write complete script, just give us test -script with following format. Please test the script and make sure -you write the correct ACTUAL OUTPUT and EXPECTED OUTPUT before you -submit. - -<?php -/* -Bug #12345 -substr() bug. Do not return expected string. - -ACTUAL OUTPUT -XYXA - -EXPECTED OUTPUT -ABCD -*/ - -$str = "XYZABCD"; -echo substr($str,3,7); - -?> |