path: root/README.TESTING

[IMPORTANT NOTICE]
------------------
 Failed tests usually indicate a problem with your local system setup
and not within PHP itself (at least for official PHP release versions).
You may decide to automatically submit a test summary to our QA workflow
at the end of a test run.
 Please do *not* submit a failed test as a bug or ask for help on why
it failed on your system without providing substantial backup information
on *why* the test failed on your special setup. Thank you :-)


[Testing Basics]
----------------
 The easiest way to test your PHP build is to run "make test" from the
command line after successfully compiling. This will run the complete
tests for all enabled functionalities and extensions using the PHP
CLI binary.
 To execute test scripts, you must build PHP with some SAPI, then you
type "make test" to execute all or some test scripts saved under
"tests" directory under source root directory.

Usage:
make test

 "make test" basically executes "run-tests.php" script
under the source root (parallel builds will not work). Therefore you
can execute the script as follows:

TEST_PHP_EXECUTABLE=sapi/cli/php \
sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]


[Which "php" executable "make test" look for]
---------------------------------------------
If you are running the run-tests.php script from the command line (as above)
you must set the TEST_PHP_EXECUTABLE environment variable to explicitly
select the PHP executable that is to be tested, that is, used to run the test scripts.

If you run the tests using make test, the PHP CLI and CGI executables are
automatically set for you. "make test" executes "run-tests.php" script with the CLI binary.  Some
test scripts such as session must be executed by CGI SAPI. Therefore,
you must build PHP with CGI SAPI to perform all tests.

NOTE: PHP binary executing "run-tests.php" and php binary used for
executing test scripts may differ. If you use different PHP binary for
executing "run-tests.php" script, you may get errors.


[Which php.ini is used]
-----------------------
 "make test" uses the same php.ini file as it would once installed.
The tests have been written to be independent of that php.ini file,
so if you find a test that is affected by a setting, please report
this, so we can address the issue.


[Which test scripts are executed]
---------------------------------
 "run-tests.php" ("make test"), without any arguments executes all
test scripts by extracting all directories named "tests"
from the source root and any subdirectories below. If there are files,
which have a "phpt" extension, "run-tests.php" looks at the sections
in these files, determines whether it should run it, by evaluating
the 'SKIPIF' section. If the test is eligible for execution, the 'FILE'
section is extracted into a ".php" file (with the same name besides
the extension) and gets executed.
When an argument is given or TESTS environment variable is set, the
GLOB is expanded by the shell and any file with extension "*.phpt" is
regarded as a test file.

 Tester can easily execute tests selectively with as follows.

Examples:
./sapi/cli/php run-tests.php ext/mbstring/*
./sapi/cli/php run-tests.php ext/mbstring/020.phpt


[Test results]
--------------
 Test results are printed to standard output. If there is a failed test,
the "run-tests.php" script saves the result, the expected result and the
code executed to the test script directory. For example, if
ext/myext/tests/myext.phpt fails to pass, the following files are created:

ext/myext/tests/myext.php   - actual test file executed
ext/myext/tests/myext.log   - log of test execution (L)
ext/myext/tests/myext.exp   - expected output (E)
ext/myext/tests/myext.out   - output from test script (O)
ext/myext/tests/myext.diff  - diff of .out and .exp (D)

 Failed tests are always bugs. Either the test is bugged or not considering
factors applying to the tester's environment, or there is a bug in PHP.
If this is a known bug, we strive to provide bug numbers, in either the
test name or the file name. You can check the status of such a bug, by
going to: http://bugs.php.net/12345 where 12345 is the bug number.
For clarity and automated processing, bug numbers are prefixed by a hash
sign '#' in test names and/or test cases are named bug12345.phpt.

NOTE: The files generated by tests can be selected by setting the
environment variable TEST_PHP_LOG_FORMAT. For each file you want to be
generated use the character in brackets as shown above (default is LEOD).
The php file will be generated always.

NOTE: You can set environment variable TEST_PHP_DETAILED to enable
detailed test information.

[Automated Testing]
--------------
 If you like to keep up to speed, with latest developments and quality
assurance, setting the environment variable NO_INTERACTION to 1, will not
prompt the tester for any user input.

Normally, the exit status of "make test" is zero, regardless of the results
of independent tests. Set the environment variable REPORT_EXIT_STATUS to 1,
and "make test" will set the exit status ("$?") to non-zero, when an
individual test has failed.

Example script to be run by cron(1):
========== qa-test.sh =============
#!/bin/sh

CO_DIR=$HOME/cvs/php7
MYMAIL=qa-test@domain.com
TMPDIR=/var/tmp
TODAY=`date +"%Y%m%d"`

# Make sure compilation environment is correct
CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre'
export MAKE=gmake
export CC=gcc

# Set test environment
export NO_INTERACTION=1
export REPORT_EXIT_STATUS=1

cd $CO_DIR
cvs update . >>$TMPDIR/phpqatest.$TODAY
./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE
$MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1
if test $? -gt 0
then
        cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL
fi
========== end of qa-test.sh =============

NOTE: The exit status of run-tests.php will be 1 when
REPORT_EXIT_STATUS is set. The result of "make test" may be higher
than that. At present, gmake 3.79.1 returns 2, so it is
advised to test for non-zero, rather then a specific value.


[Creating new test files]
-------------------------
 Writing test file is very easy if you are used to PHP.
See the HOWTO at http://qa.php.net/write-test.php


[How to help us]
----------------
 If you find bug in PHP, you can submit bug report AND test script
for us. You don't have to write complete script, just give us test
script with following format. Please test the script and make sure
you write the correct ACTUAL OUTPUT and EXPECTED OUTPUT before you
submit.

<?php
/*
Bug #12345
substr() bug. Do not return expected string.

ACTUAL OUTPUT
XYXA

EXPECTED OUTPUT
ABCD
*/

$str = "XYZABCD";
echo substr($str,3,7);

?>

[IMPORTANT NOTICE]
------------------
This is an addendum to README.TESTING with additional information
specific to server-tests.php.

server-tests.php is backward compatible with tests developed for
the original run-tests.php script.  server-tests is *not* used by
'make test'.  server-tests was developed to provide support for
testing PHP under it's primary environment, HTTP, and can run the
PHP tests under any of the SAPI modules that are direct executables,
or are accessible via HTTP.

[New features]
----------------
* Command line interface:
  You can run 'php server-tests.php -h' to get all the possible options.
* Configuration file:
  the -c argument will allow you to use a configuration file.  This is
  handy if you are testing multiple environments and need various options
  depending on the environment.
  see server-tests-config.php for details.
* CGI Emulation:
  Will emulate a CGI environment when testing with the cgi sapi executable.
* HTTP testing:
  can be configured to run test scripts through an HTTP server running
  on localhost.  localhost is required since either the web server must
  alias a directory to the php source directory, or the test scripts
  must be copied to a directory under the web server
  (see config options TEST_WEB_BASE_URL, TEST_BASE_PATH, and TEST_WEB_EXT)
* New sections supported for test files (see below)

When running tests over http, tests that require ini settings different that what
the web server runs under will be skipped.  Since the test harness defines a number
of ini settings by default, the web server may require special configuration to
make testing work.

[Example Usage]
----------------
Some (but not all!) examples of usage:

1. run tests from the php source directory
    php server-tests.php -p /path/to/php-cli

2. run tests using cgi emulation
    php server-tests.php -p /path/to/php-cgi

3. run tests over http, copying test files into document root
    php server-tests.php -w -u http://localhost/test -m /path/to/htdocs/test

4. run tests over http, php sources have been aliased in web server
    php server-tests.php -w -u http://localhost/test

5. run tests using configuration file
    php server-tests.php -c /path/to/server-tests-config.php

6. run tests using configuration file, but overriding some settings:
   (config file must be first)
    php server-tests.php -c /path/to/server-tests-config.php -w -t 3 -d /path/to/testdir

NOTE: configuration as described in README.TESTING still works.

[New Test Sections]
----------------
In addition to the traditional test sections
(see http://qa.php.net/write-test.php), several new sections are available
under server-tests.

--POST--
This is not a new section, but not multipart posts are supported for testing
file uploads, or other types of POST data.

--CGI--
This section takes no value.  It merely provides a simple marker for tests
that MUST be run as CGI, even if there is no --POST-- or --GET-- sections
in the test file.

--DESCRIPTION--
Not used for anything, just a section for documenting the test

--ENV--
This section get's eval()'d to help build an environment for the
execution of the test.  This can be used to change environment
vars that are used for CGI emulation, or simply to set env vars
for cli testing.  A full example looks like:

  --ENV--
  return <<<END
  PATH_TRANSLATED=$filename
  PATH_INFO=$scriptname
  SCRIPT_NAME=$scriptname
  END;

Some variables are made easily available for use in this section, they
include:
  $filename     full native path to file, will become PATH_TRANSLATED
  $filepath     =dirname($filename)
  $scriptname   this is what will become SCRIPT_NAME unless you override it
  $docroot      the equivalent of DOCUMENT_ROOT under Apache
  $cwd          the directory that the test is being initiated from
  $this->conf   all server-tests configuration vars
  $this->env    all environment variables that will get passed to the test


--REQUEST--
This section is also eval'd, and is similar in nature to --ENV--.  However,
this section is used to build the url used in an HTTP request.  Valid values
to set in this section would include:
  SCRIPT_NAME   The initial part of the request url
  PATH_INFO     The pathinfo part of a request url
  FRAGMENT      The fragment section of a url (after #)
  QUERY_STRING  The query part of a url (after ?)

  --REQUEST--
  return <<<END
  PATH_INFO=/path/info
  END;

--HEADERS--
This section is also eval'd.  It is used to provide additional headers sent
in an HTTP request, such as content type for multipart posts, cookies, etc.

  --HEADERS--
  return <<<END
  Content-Type=multipart/form-data; boundary=---------------------------240723202011929
  Content-Length=100
  END;

--EXPECTHEADERS--
This section can be used to define what headers are required to be
received back from a request, and is checked in addition to the
regular expect sections.  For example:

  --EXPECTHEADERS--
  Status: 404