diff options
Diffstat (limited to 'test/README')
| -rw-r--r-- | test/README | 381 |
1 files changed, 214 insertions, 167 deletions
diff --git a/test/README b/test/README index ea775cfcd..e8b5ff6cc 100644 --- a/test/README +++ b/test/README @@ -1,108 +1,168 @@ -Regression test suite for cairo. - How to use cairo's test suite ============================= Using this test should be as simple as running: - make test + meson test -C $builddir -v assuming that the cairo distribution in the directory above has been -configured and built. The test suite here goes through some effort to -run against the locally compiled library rather than any installed -version, but those efforts may fall short depending on the level of your -libtool madness. +configured and built in $builddir. The test suite here will use the +locally compiled library rather than installed version. + +The above command runs some source code as well as the rendering +tests. All the rendering tests are in a single meson test named +"cairo". It can be run with: -The results of the test suite run are summarized in an index.html -file, which, when viewed in a web browser makes it quite easy to -visually see any failed renderings alongside the corresponding -reference image, (and a diff image as well). + meson test -C $builddir -v cairo + +The results of the test suite run are summarized in an index.html file +written to $builddir, which, when viewed in a web browser makes it +quite easy to visually see any failed renderings alongside the +corresponding reference image, (and a diff image as well). As some browsers do not permit Javascript to read from the local -filesystem, the view-test-results.py script can used to view the -results. It starts a http server serving the current directory before -displaying the test results in a browser. +filesystem, the view-test-results.py script in the $builddir can used +to view the results. It starts a http server serving the current +directory before displaying the test results in a browser. The test suite needs to be run before any code is committed and before any release. See below for hints and rules governing the use of the suite. -The test suite is built as a single binary, which allows you to choose -individual or categories of tests to run. For example, to run specific tests: - ./cairo-test-suite record-neg-extents-unbounded record-neg-extents-bounded -Or if you want to run all paint.* related tests you can use: - ./cairo-test-suite paint -Or if you want to check the current status of known failures: - ./cairo-test-suite XFAIL -Or to run a subset of tests, use the -k option to run only the tests -that include the given keyword: - ./cairo-test-suite -k downscale -The binary also permits controlling which backend is used via the -CAIRO_TEST_TARGET environment variable, so for instance: - CAIRO_TEST_TARGET=gl ./cairo-test-suite -k blur -This binary should be backwards-compatible with all library versions, -allowing you to compare current versus past behaviour for any test. - -The test suite needs to find the "test" directory in the source -tree. The srcdir environment variable can be used to specify -the location of this directory. If this environment variable is not -set, the binary looks for the directory "srcdir" in the current -directory. The meson build system symlinks "srcdir" in the -$builddir/test directory to the "test" directory in the source -tree. If this is not found the binary defaults to the current -directory. - -Tailoring tests running ------------------------ -There are some mechanisms to limit the tests run during "make test". -These come very handy when doing development, but should not be used -to circumvent the "pass" requirements listed below. +Running specific tests +---------------------- +During development it is desirable to only run a single test or groups +of related tests. Individual tests can be run by specifying the tests +names as arguments to the cairo test. + + meson test -v cairo --test-args "record-neg-extents-unbounded record-neg-extents-bounded" -make's TARGETS environment variable can be used to limit the backends when -running the tests. It should contain a (space-, comma-separated) list of -backends. CAIRO_TESTS environment variable, which is a comma-, space-seperated -lists, can be used to limit the tests run. -For example: +The test suite is built as a single binary, "cairo-test-suite", which +can be run directly instead of via meson. This may be more convenient +when specifying multiple test arguments. The executable needs to be +run in $builddir/test in order to find the image conversion +executables. Simply running the executable will run all tests. eg - CAIRO_TESTS="zero-alpha" make test TARGETS=image,ps + cd $builddir/test + ./cairo-test-suite -make's FORMAT variable can also be used to limit the content formats when -running the tests. It should contain a (space-, comma-separated) list of -content formats to test. -For example: +Normally the executable will use the locally compiled library. But if +you have a LD_LIBRARY_PATH defined or built cairo with an -rpath, the +executable may not link with the locally compiled library. - CAIRO_TESTS="zero-alpha" make test TARGETS=image,ps FORMAT="rgb,rgba" +Individual tests can be run be specifying the tests names as arguments +to cairo-test-suite. For example, to run specific tests: -Another very handy mechanism when trying to fix bugs is: + ./cairo-test-suite record-neg-extents-unbounded record-neg-extents-bounded - make retest +Or if you want to run all paint.* related tests you can use: -This will re-run the test suite, but only on tests that failed on the -last run. So this is a much faster way of checking if changes actually -fix bugs rather than running the entire test suite again. + ./cairo-test-suite paint -The test suite first compares the output from the current run against the -previous in order to skip more expensive image comparisons . If you think -this is interfering with the results, you can clear the cached results using: +Or if you want to check the current status of known failures: - make clean-caches + ./cairo-test-suite XFAIL -Running tests under modified environments or tools -------------------------------------------------- -To run tests under a tool like gdb, one can use the run target and -the TOOL variable. For example: +Or to run a subset of tests, use the -k option to run only the tests +that include the given keyword: + + ./cairo-test-suite -k downscale - CAIRO_TESTS=user-font make run TOOL=gdb TARGETS=pdf +The -l option lists all tests available. For convenience, the file +"completion.bash" in $builddir can be sourced to provide bash +completion of the test names in cairo-test-suite. -If you want to run under valgrind, there is a specific target for that -that also sets a bunch of useful valgrind options. Try: +By default, each test that is run will be tested on all backends that +have been built. The CAIRO_TEST_TARGET environment variable can be +used to select specific backends. For example: - CAIRO_TESTS=user-font make check-valgrind + CAIRO_TEST_TARGET=pdf ./cairo-test-suite text-rotate -You can run tests under a modified environment you can use the ENV -make variable. However, that environment will also affect the libtool -wrapper of the tests. To only affect the actual test binaries, pass -such environment as TOOL: +Each test is run twice on each backend, once for each format. The +CAIRO_TEST_TARGET_FORMAT environment variable selects a single +format. The formats are "rgba" and "rgb". For example to run the +text-rotate test only on the image backend and only for the "rgba" +format: - CAIRO_TESTS=user-font make run TOOL="LD_PRELOAD=/path/to/something.so" + CAIRO_TEST_TARGET=image CAIRO_TEST_TARGET_FORMAT=rgba ./cairo-test-suite text-rotate + +Running tests under modified environments or tools +-------------------------------------------------- +To run tests under in gdb, one can simply run the cairo-suite-suite in +gdb. Usually a specific test will be desired. eg + + CAIRO_TEST_TARGET=image CAIRO_TEST_TARGET_FORMAT=rgba gdb --args ./cairo-test-suite text-rotate + +The cairo test suite runs each test in a child process so that crashes +do not stop the test. When running the test suite under tools, the "-f" +option disables forking. eg to run a test under valgrind: + + CAIRO_TEST_TARGET=image CAIRO_TEST_TARGET_FORMAT=rgba valgrind ./cairo-test-suite -f text-rotate + +The cairo test suite detects if it is running under gdb and disables +forking. Hence the reason the gdb example above did not use "-f". + +Complete list of test options +----------------------------- +cairo-test-suite options: + + ./cairo-test-suite [-afkxsl] [test-names|keywords ...] + + -a all; run the full set of tests. By default the test suite + skips similar surface and device offset testing. + -f foreground; do not fork + -k match tests by keyword + -l list only; just list selected test case names without executing + -s include slow, long running tests + -x exit on first failure + +Environment Variables: + +CAIRO_TEST_TARGET + Run the test only the the listed targets. eg + CAIRO_TEST_TARGET="image" + CAIRO_TEST_TARGET="pdf,ps3,ps2" + +CAIRO_TEST_TARGET_FORMAT + Run the test only the the listed targets. eg + CAIRO_TEST_TARGET_FORMAT="rgba" + CAIRO_TEST_TARGET_FORMAT="rgba,rgb" + +CAIRO_TEST_MODE + May contain a list of one or more of the following options + "full" - equivalent to the "-a" option + "similar" + "offset" + "scale" + "foreground" - equivalent to the "-f" option + "exit-on-failure" - equivalent to the "-x" option + eg + CAIRO_TEST_MODE="full,foreground" + +CAIRO_TESTS + A list of test to run. This is equivalent to listing the tests as + arguments to cairo=test-suite. eg + CAIRO_TESTS="record-neg-extents-unbounded record-neg-extents-bounded" + The tests may be separated by any of " \t,:;". + +srcdir + The test suite needs to find the "test" directory in the source + tree. The srcdir environment variable can be used to specify the + location of this directory. If this environment variable is not set, + the binary looks for the directory "srcdir" in the current + directory. The meson build system symlinks "srcdir" in the + $builddir/test directory to the "test" directory in the source + tree. If this is not found the binary defaults to the current + directory. + +CAIRO_REF_DIR + The directory containing the reference images. By default the test + suite looks for "reference" in $srcdir. For convenience, you can set + CAIRO_REF_DIR to point at a previous test directory, relative to the + current test directory, and any previous output will be used by + preference as reference images. + +CAIRO_TEST_TIMEOUT + The maximum time, in seconds, an indivdual test may run before it is + stopped. The default is 60 seconds. Getting the elusive zero failures --------------------------------- @@ -115,55 +175,10 @@ which the reference images were generated, then you will likely see the test suite reporting "failures", (even if cairo is working just fine). -We are constantly working to reduce the number of variables that need -to be tweaked to get a clean run, (for example, by bundling fonts with -the test suite itself), and also working to more carefully document -the software configuration used to generate the reference images. - -Here are some of the relevant details: - - * Your system must have a copy of the DejaVu font, the sha1sum of - the version used are listed in [...]. These are - "DejaVu Sans" (DejaVuSans.ttf) [e9831ee4fd2e1d0ac54508a548c6a449545eba3f]; - "DejaVu Sans Mono" (DejaVuSansMono.ttf) [25d854fbd0450a372615a26a8ef9a1024bd3efc6]; - "DejaVu Serif" (DejaVuSerif.ttf) [78a81850dc7883969042cf3d6dfd18eea7e43e2f]; - [the DejaVu fonts can be installed from the fonts-dejavu-core 2.34-1 Debian package] - and also - "Nimbus Sans L" (n019003l.pfb) - [which can be found in the gsfonts Debian package]. - - * Currently, you must be using a build of cairo using freetype - (cairo-ft) as the default font backend. Otherwise all tests - involving text are likely to fail. - - * To test the pdf backend, you will want the very latest version of - poppler as made available via git: - - git clone git://anongit.freedesktop.org/git/poppler/poppler - - As of this writing, no released version of poppler contains all - the fixes you will need to avoid false negatives from the test - suite. - - * To test the ps backend, you will need ghostscript version 9.06. - - * Testing the xlib backend is problematic since many X server - drivers have bugs that are exercised by the test suite. (Or, if - not actual bugs, differ slightly in their output in such a way - that the test suite will report errors.) This can be quite handy - if you want to debug an X server driver, but since most people - don't want to do that, another option is to run against a headless - X server that uses only software for all rendering. One such X - server is Xvfb which can be started like this: - - Xvfb -screen 0 1680x1024x24 -ac -nolisten tcp :2 - - after which the test suite can be run against it like so: - - DISPLAY=:2 make test - - We have been using Xvfb for testing cairo releases and ensuring - that all tests behave as expected with this X server. +Since the test suite must pass the Continuous Integration test in the +Gitlab instance hosting the cairo repository, the easiest way to get a +new or updated test to pass is to extract the test output from the +cairo gitlab CI to use as reference images. What if I can't make my system match? ------------------------------------- @@ -179,17 +194,18 @@ suite before your changes, and then use the CAIRO_REF_DIR environment variable to use that output as the reference images for a run after your changes. The process looks like this: - # Before code change there may be failures we don't care about - make test + # Before code change there may be failures we don't care about + cd $builddir/test + ./cairo-test-suite - # Let's save those output images - mkdir /some/directory/ - cp -r test/output /some/directory/ + # Let's save those output images + mkdir /some/directory/ + cp -r output /some/directory/ - # hack, hack, hack + # hack, hack, hack - # Now to see if nothing changed: - CAIRO_REF_DIR=/some/directory/ make test + # Now to see if nothing changed: + CAIRO_REF_DIR=/some/directory/ ./cairo-test-suite Best practices for cairo developers =================================== @@ -197,16 +213,67 @@ If we all follow the guidelines below, then both the test suite and cairo itself will stay much healthier, and we'll all have a lot more fun hacking on cairo. -Before committing ------------------ -All tests should return a result of PASS or XFAIL. The XFAIL results -indicate known bugs. The final message should be one of the following: +Before creating a Merge Request +------------------------------- +After pushing your branch to your cairo fork on gitlab.freedesktop.org, +check that the CI passes. A MR cannot be merged until the CI passes. - All XX tests behaved as expected (YY expected failures) - All XX tests passed +It is likely that you will need to download test output images from +the Gitlab CI to use as reference images in order to get CI to pass. + +The reference image filenames have the form: + +<test name>.*.ref.png for correct images that pass. + +<test name>.*.xfail.png for incorrect images due to bugs. + +The <test name>.ref.png is the most general image that all backends +and formats for the <test name> test will be required to match. + +When comparing test output to the reference images, the test suite +performs a perceptual diff which permits some minor pixel differences +that are not visually discernible. Some targets may have differences +that exceed the threshold of the perceptual diff but still appear the +same as the reference image when eyeballed. In these cases, target +specific reference images can be created that override the general +reference image for the specific target. + +Reference images for specific backends can be created using filenames of the form: + <test name>.<target>.ref.png + +eg clip.pdf.ref.png + +The target name "ps" can be used for both "ps2" and "ps3" targets. + +Reference images for specific formats can be created using filenames of the form: + <test name>.<target>.<format>.ref.png -If any tests have a status of FAIL, then the new code has caused a -regression error which should be fixed before the code is committed. +eg + clip.pdf.argb32.ref.png + clip.pdf.rgb24.ref.png + +The recommended practice for creating reference images for a new test is: + +1. Create a <test name>.ref.png reference image from <test name>.image.argb32.out.png +output. + +2. Push to gitlab and check the CI result. + +3. If the image target fails, copy the <test name>.image.argb32.out.png output +from CI and use as the reference image. + +4. Once the image target passes in CI, check that the other targets +pass. If any fail, create <test name>.<target>.ref.png reference images if the +output appears correct. + +5. If a target output is incorrect, try to determine if the cairo +target output is correct. eg view pdf files in Adobe Reader. If the +cairo target output is correct, it can be assumed the image converson +from target to png is buggy. File a bug against the conversion tool +and create a <test name>.<target>.xfail.png reference image. + +In all cases, before creating a reference image from test output, the +image should be viewed to ensure that it is correct. When a new bug is found ----------------------- @@ -222,7 +289,7 @@ the style of existing tests. The new-bug.ref.png image should contain the desired result of new-bug.c if the bug were fixed while new-bug.xfail.png contains the current results of the test. -Makefile.sources should be edited by adding new-bug.c to test_sources. +"meson.build" should be edited by adding new-bug.c to test_sources. And last but not least, don't forget to "git add" the new files. When a new feature is added @@ -263,34 +330,14 @@ PostScript, and SVG), into an image that can be used for the image comparison. This means that any bugs in that conversion tool will result in false negatives in the test suite. -We've identified several such bugs in the poppler library which is -used to convert PDF to an image. This is particularly discouraging -because 1) poppler is free software that will be used by *many* cairo -users, and 2) poppler calls into cairo for its rendering so it should -be able to do a 100% faithful conversion. - -So we have an interest in ensuring that these poppler bugs get fixed -sooner rather than later. As such, we're trying to be good citizens by -reporting all such poppler bugs that we identify to the poppler -bugzilla. Here's a tracking bug explaining the situation: - - Poppler does not yet handle everything in the cairo test suite - https://bugs.freedesktop.org/show_bug.cgi?id=12143 - Here's the rule: If a cairo-pdf test reports a failure, but viewing the resulting PDF file with acroread suggests that the PDF itself is correct, then there's likely a bug in poppler. In this case, we can -simply report the poppler bug, (making it block 12143 above), post the -PDF result from the test suite, and list the bug in this file. Once -we've done this, we can capture poppler's buggy output as a -pdf-specific reference image (as reference/*.xfail.png) so that the -test suite will regard the test as passing, (and we'll ensure there -is no regression). +simply report the poppler bug. Once we've done this, we can capture +poppler's buggy output as a pdf-specific reference image (as +reference/*.xfail.png) so that the test suite will regard the test as +passing, (and we'll ensure there is no regression). Once the poppler bug gets fixed, the test suite will start reporting a false negative again, and this will be easy to fix by simply removing the pdf-specific reference image. - -Here are the reported poppler bugs and the tests they affect: - -[Newest was closed in 2009.] |