Core Git Tests ============== This directory holds many test scripts for core Git tools. The first part of this short document describes how to run the tests and read their output. When fixing the tools or adding enhancements, you are strongly encouraged to add tests in this directory to cover what you are trying to fix or enhance. The later part of this short document describes how your test scripts should be organized. Running Tests ------------- The easiest way to run tests is to say "make". This runs all the tests. *** t0000-basic.sh *** ok 1 - .git/objects should be empty after git init in an empty repo. ok 2 - .git/objects should have 3 subdirectories. ok 3 - success is reported like this ... ok 43 - very long name in the index handled sanely # fixed 1 known breakage(s) # still have 1 known breakage(s) # passed all remaining 42 test(s) 1..43 *** t0001-init.sh *** ok 1 - plain ok 2 - plain with GIT_WORK_TREE ok 3 - plain bare Since the tests all output TAP (see http://testanything.org) they can be run with any TAP harness. Here's an example of parallel testing powered by a recent version of prove(1): $ prove --timer --jobs 15 ./t[0-9]*.sh [19:17:33] ./t0005-signals.sh ................................... ok 36 ms [19:17:33] ./t0022-crlf-rename.sh ............................... ok 69 ms [19:17:33] ./t0024-crlf-archive.sh .............................. ok 154 ms [19:17:33] ./t0004-unwritable.sh ................................ ok 289 ms [19:17:33] ./t0002-gitfile.sh ................................... ok 480 ms ===( 102;0 25/? 6/? 5/? 16/? 1/? 4/? 2/? 1/? 3/? 1... )=== prove and other harnesses come with a lot of useful options. The --state option in particular is very useful: # Repeat until no more failures $ prove -j 15 --state=failed,save ./t[0-9]*.sh You can give DEFAULT_TEST_TARGET=prove on the make command (or define it in config.mak) to cause "make test" to run tests under prove. GIT_PROVE_OPTS can be used to pass additional options, e.g. $ make DEFAULT_TEST_TARGET=prove GIT_PROVE_OPTS='--timer --jobs 16' test You can also run each test individually from command line, like this: $ sh ./t3010-ls-files-killed-modified.sh ok 1 - git update-index --add to add various paths. ok 2 - git ls-files -k to show killed files. ok 3 - validate git ls-files -k output. ok 4 - git ls-files -m to show modified files. ok 5 - validate git ls-files -m output. # passed all 5 test(s) 1..5 You can pass --verbose (or -v), --debug (or -d), and --immediate (or -i) command line argument to the test, or by setting GIT_TEST_OPTS appropriately before running "make". Short options can be bundled, i.e. '-d -v' is the same as '-dv'. -v:: --verbose:: This makes the test more verbose. Specifically, the command being run and their output if any are also output. --verbose-only=:: Like --verbose, but the effect is limited to tests with numbers matching . The number matched against is simply the running count of the test within the file. -x:: Turn on shell tracing (i.e., `set -x`) during the tests themselves. Implies `--verbose`. Ignored in test scripts that set the variable 'test_untraceable' to a non-empty value, unless it's run with a Bash version supporting BASH_XTRACEFD, i.e. v4.1 or later. -d:: --debug:: This may help the person who is developing a new test. It causes the command defined with test_debug to run. The "trash" directory (used to store all temporary data during testing) is not deleted even if there are no failed tests so that you can inspect its contents after the test finished. -i:: --immediate:: This causes the test to immediately exit upon the first failed test. Cleanup commands requested with test_when_finished are not executed if the test failed, in order to keep the state for inspection by the tester to diagnose the bug. -l:: --long-tests:: This causes additional long-running tests to be run (where available), for more exhaustive testing. -r:: --run=:: Run only the subset of tests indicated by . See section "Skipping Tests" below for syntax. --valgrind=:: Execute all Git binaries under valgrind tool and exit with status 126 on errors (just like regular tests, this will only stop the test script when running under -i). Since it makes no sense to run the tests with --valgrind and not see any output, this option implies --verbose. For convenience, it also implies --tee. defaults to 'memcheck', just like valgrind itself. Other particularly useful choices include 'helgrind' and 'drd', but you may use any tool recognized by your valgrind installation. As a special case, can be 'memcheck-fast', which uses memcheck but disables --track-origins. Use this if you are running tests in bulk, to see if there are _any_ memory issues. Note that memcheck is run with the option --leak-check=no, as the git process is short-lived and some errors are not interesting. In order to run a single command under the same conditions manually, you should set GIT_VALGRIND to point to the 't/valgrind/' directory and use the commands under 't/valgrind/bin/'. --valgrind-only=:: Like --valgrind, but the effect is limited to tests with numbers matching . The number matched against is simply the running count of the test within the file. --tee:: In addition to printing the test output to the terminal, write it to files named 't/test-results/$TEST_NAME.out'. As the names depend on the tests' file names, it is safe to run the tests with this option in parallel. -V:: --verbose-log:: Write verbose output to the same logfile as `--tee`, but do _not_ write it to stdout. Unlike `--tee --verbose`, this option is safe to use when stdout is being consumed by a TAP parser like `prove`. Implies `--tee` and `--verbose`. --with-dashes:: By default tests are run without dashed forms of commands (like git-commit) in the PATH (it only uses wrappers from ../bin-wrappers). Use this option to include the build directory (..) in the PATH, which contains all the dashed forms of commands. This option is currently implied by other options like --valgrind and GIT_TEST_INSTALLED. --no-bin-wrappers:: By default, the test suite uses the wrappers in `../bin-wrappers/` to execute `git` and friends. With this option, `../git` and friends are run directly. This is not recommended in general, as the wrappers contain safeguards to ensure that no files from an installed Git are used, but can speed up test runs especially on platforms where running shell scripts is expensive (most notably, Windows). --root=:: Create "trash" directories used to store all temporary data during testing under , instead of the t/ directory. Using this option with a RAM-based filesystem (such as tmpfs) can massively speed up the test suite. --chain-lint:: --no-chain-lint:: If --chain-lint is enabled, the test harness will check each test to make sure that it properly "&&-chains" all commands (so that a failure in the middle does not go unnoticed by the final exit code of the test). This check is performed in addition to running the tests themselves. You may also enable or disable this feature by setting the GIT_TEST_CHAIN_LINT environment variable to "1" or "0", respectively. --stress:: Run the test script repeatedly in multiple parallel jobs until one of them fails. Useful for reproducing rare failures in flaky tests. The number of parallel jobs is, in order of precedence: the value of the GIT_TEST_STRESS_LOAD environment variable, or twice the number of available processors (as shown by the 'getconf' utility), or 8. Implies `--verbose -x --immediate` to get the most information about the failure. Note that the verbose output of each test job is saved to 't/test-results/$TEST_NAME.stress-.out', and only the output of the failed test job is shown on the terminal. The names of the trash directories get a '.stress-' suffix, and the trash directory of the failed test job is renamed to end with a '.stress-failed' suffix. --stress-jobs=:: Override the number of parallel jobs. Implies `--stress`. --stress-limit=:: When combined with --stress run the test script repeatedly this many times in each of the parallel jobs or until one of them fails, whichever comes first. Implies `--stress`. You can also set the GIT_TEST_INSTALLED environment variable to the bindir of an existing git installation to test that installation. You still need to have built this git sandbox, from which various test-* support programs, templates, and perl libraries are used. If your installed git is incomplete, it will silently test parts of your built version instead. When using GIT_TEST_INSTALLED, you can also set GIT_TEST_EXEC_PATH to override the location of the dashed-form subcommands (what GIT_EXEC_PATH would be used for during normal operation). GIT_TEST_EXEC_PATH defaults to `$GIT_TEST_INSTALLED/git --exec-path`. Skipping Tests -------------- In some environments, certain tests have no way of succeeding due to platform limitation, such as lack of 'unzip' program, or filesystem that do not allow arbitrary sequence of non-NUL bytes as pathnames. You should be able to say something like $ GIT_SKIP_TESTS=t9200.8 sh ./t9200-git-cvsexport-commit.sh and even: $ GIT_SKIP_TESTS='t[0-4]??? t91?? t9200.8' make to omit such tests. The value of the environment variable is a SP separated list of patterns that tells which tests to skip, and either can match the "t[0-9]{4}" part to skip the whole test, or t[0-9]{4} followed by ".$number" to say which particular test to skip. For an individual test suite --run could be used to specify that only some tests should be run or that some tests should be excluded from a run. The argument for --run, , is a list of description substrings or globs or individual test numbers or ranges with an optional negation prefix (of '!') that define what tests in a test suite to include (or exclude, if negated) in the run. A range is two numbers separated with a dash and matches a range of tests with both ends been included. You may omit the first or the second number to mean "from the first test" or "up to the very last test" respectively. The argument to --run is split on commas into separate strings, numbers, and ranges, and picks all tests that match any of the individual selection criteria. If the substring of the description text that you want to match includes a comma, use the glob character '?' instead. For example --run='rebase,merge?cherry-pick' would match on all tests that match either the glob *rebase* or the glob *merge?cherry-pick*. If --run starts with an unprefixed number or range the initial set of tests to run is empty. If the first item starts with '!' all the tests are added to the initial set. After initial set is determined every test number or range is added or excluded from the set one by one, from left to right. For example, to run only tests up to a specific test (21), one could do this: $ sh ./t9200-git-cvsexport-commit.sh --run='1-21' or this: $ sh ./t9200-git-cvsexport-commit.sh --run='-21' Common case is to run several setup tests (1, 2, 3) and then a specific test (21) that relies on that setup: $ sh ./t9200-git-cvsexport-commit.sh --run='1,2,3,21' or: $ sh ./t9200-git-cvsexport-commit.sh --run=1,2,3,21 or: $ sh ./t9200-git-cvsexport-commit.sh --run='-3,21' As noted above, the test set is built by going through the items from left to right, so this: $ sh ./t9200-git-cvsexport-commit.sh --run='1-4,!3' will run tests 1, 2, and 4. Items that come later have higher precedence. It means that this: $ sh ./t9200-git-cvsexport-commit.sh --run='!3,1-4' would just run tests from 1 to 4, including 3. You may use negation with ranges. The following will run all test in the test suite except from 7 up to 11: $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11' Sometimes there may be multiple tests with e.g. "setup" in their name that are needed and rather than figuring out the number for all of them we can just use "setup" as a substring/glob to match against the test description: $ sh ./t0050-filesystem.sh --run=setup,9-11 or one could select both the setup tests and the rename ones (assuming all relevant tests had those words in their descriptions): $ sh ./t0050-filesystem.sh --run=setup,rename Some tests in a test suite rely on the previous tests performing certain actions, specifically some tests are designated as "setup" test, so you cannot _arbitrarily_ disable one test and expect the rest to function correctly. --run is mostly useful when you want to focus on a specific test and know what setup is needed for it. Or when you want to run everything up to a certain test. Running tests with special setups --------------------------------- The whole test suite could be run to test some special features that cannot be easily covered by a few specific test cases. These could be enabled by running the test suite with correct GIT_TEST_ environment set. GIT_TEST_FAIL_PREREQS= fails all prerequisites. This is useful for discovering issues with the tests where say a later test implicitly depends on an optional earlier test. There's a "FAIL_PREREQS" prerequisite that can be used to test for whether this mode is active, and e.g. skip some tests that are hard to refactor to deal with it. The "SYMLINKS" prerequisite is currently excluded as so much relies on it, but this might change in the future. GIT_TEST_SPLIT_INDEX= forces split-index mode on the whole test suite. Accept any boolean values that are accepted by git-config. GIT_TEST_PASSING_SANITIZE_LEAK=true skips those tests that haven't declared themselves as leak-free by setting "TEST_PASSES_SANITIZE_LEAK=true" before sourcing "test-lib.sh". This test mode is used by the "linux-leaks" CI target. GIT_TEST_PASSING_SANITIZE_LEAK=check checks that our "TEST_PASSES_SANITIZE_LEAK=true" markings are current. Rather than skipping those tests that haven't set "TEST_PASSES_SANITIZE_LEAK=true" before sourcing "test-lib.sh" this mode runs them with "--invert-exit-code". This is used to check that there's a one-to-one mapping between "TEST_PASSES_SANITIZE_LEAK=true" and those tests that pass under "SANITIZE=leak". This is especially useful when testing a series that fixes various memory leaks with "git rebase -x". GIT_TEST_SANITIZE_LEAK_LOG=true will log memory leaks to "test-results/$TEST_NAME.leak/trace.*" files. The logs include a "dedup_token" (see +"ASAN_OPTIONS=help=1 ./git") and other options to make logs +machine-readable. With GIT_TEST_SANITIZE_LEAK_LOG=true we'll look at the leak logs before exiting and exit on failure if the logs showed that we had a memory leak, even if the test itself would have otherwise passed. This allows us to catch e.g. missing &&-chaining. This is especially useful when combined with "GIT_TEST_PASSING_SANITIZE_LEAK", see below. GIT_TEST_PASSING_SANITIZE_LEAK=check when combined with "--immediate" will run to completion faster, and result in the same failing tests. The only practical reason to run GIT_TEST_PASSING_SANITIZE_LEAK=check without "--immediate" is to combine it with "GIT_TEST_SANITIZE_LEAK_LOG=true". If we stop at the first failing test case our leak logs won't show subsequent leaks we might have run into. GIT_TEST_PASSING_SANITIZE_LEAK=(true|check) will not catch all memory leaks unless combined with GIT_TEST_SANITIZE_LEAK_LOG=true. Some tests run "git" (or "test-tool" etc.) without properly checking the exit code, or git will invoke itself and fail to ferry the abort() exit code to the original caller. When the two modes are combined we'll look at the "test-results/$TEST_NAME.leak/trace.*" files at the end of the test run to see if had memory leaks which the test itself didn't catch. GIT_TEST_PROTOCOL_VERSION=, when set, makes 'protocol.version' default to n. GIT_TEST_FULL_IN_PACK_ARRAY= exercises the uncommon pack-objects code path where there are more than 1024 packs even if the actual number of packs in repository is below this limit. Accept any boolean values that are accepted by git-config. GIT_TEST_OE_SIZE= exercises the uncommon pack-objects code path where we do not cache object size in memory and read it from existing packs on demand. This normally only happens when the object size is over 2GB. This variable forces the code path on any object larger than bytes. GIT_TEST_OE_DELTA_SIZE= exercises the uncommon pack-objects code path where deltas larger than this limit require extra memory allocation for bookkeeping. GIT_TEST_VALIDATE_INDEX_CACHE_ENTRIES= checks that cache-tree records are valid when the index is written out or after a merge. This is mostly to catch missing invalidation. Default is true. GIT_TEST_COMMIT_GRAPH=, when true, forces the commit-graph to be written after every 'git commit' command, and overrides the 'core.commitGraph' setting to true. GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=, when true, forces commit-graph write to compute and write changed path Bloom filters for every 'git commit-graph write', as if the `--changed-paths` option was passed in. GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor code paths for utilizing a (hook based) file system monitor to speed up detecting new or changed files. GIT_TEST_INDEX_VERSION= exercises the index read/write code path for the index version specified. Can be set to any valid version (currently 2, 3, or 4). GIT_TEST_PACK_SPARSE= if disabled will default the pack-objects builtin to use the non-sparse object walk. This can still be overridden by the --sparse command-line argument. GIT_TEST_PRELOAD_INDEX= exercises the preload-index code path by overriding the minimum number of cache entries required per thread. GIT_TEST_INDEX_THREADS= enables exercising the multi-threaded loading of the index for the whole test suite by bypassing the default number of cache entries and thread minimums. Setting this to 1 will make the index loading single threaded. GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack- index to be written after every 'git repack' command, and overrides the 'core.multiPackIndex' setting to true. GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=, when true, sets the '--bitmap' option on all invocations of 'git multi-pack-index write', and ignores pack-objects' '--write-bitmap-index'. GIT_TEST_SIDEBAND_ALL=, when true, overrides the 'uploadpack.allowSidebandAll' setting to true, and when false, forces fetch-pack to not request sideband-all (even if the server advertises sideband-all). GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS=, when true (which is the default when running tests), errors out when an abbreviated option is used. GIT_TEST_DEFAULT_HASH= specifies which hash algorithm to use in the test scripts. Recognized values for are "sha1" and "sha256". GIT_TEST_NO_WRITE_REV_INDEX=, when true disables the 'pack.writeReverseIndex' setting. GIT_TEST_SPARSE_INDEX=, when true enables index writes to use the sparse-index format by default. GIT_TEST_CHECKOUT_WORKERS= overrides the 'checkout.workers' setting to and 'checkout.thresholdForParallelism' to 0, forcing the execution of the parallel-checkout code. GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=, when true, makes registering submodule ODBs as alternates a fatal action. Support for this environment variable can be removed once the migration to explicitly providing repositories when accessing submodule objects is complete or needs to be abandoned for whatever reason (in which case the migrated codepaths still retain their performance benefits). GIT_TEST_REQUIRE_PREREQ= allows specifying a space separated list of prereqs that are required to succeed. If a prereq in this list is triggered by a test and then fails then the whole test run will abort. This can help to make sure the expected tests are executed and not silently skipped when their dependency breaks or is simply not present in a new environment. Naming Tests ------------ The test files are named as: tNNNN-commandname-details.sh where N is a decimal digit. First digit tells the family: 0 - the absolute basics and global stuff 1 - the basic commands concerning database 2 - the basic commands concerning the working tree 3 - the other basic commands (e.g. ls-files) 4 - the diff commands 5 - the pull and exporting commands 6 - the revision tree commands (even e.g. merge-base) 7 - the porcelainish commands concerning the working tree 8 - the porcelainish commands concerning forensics 9 - the git tools Second digit tells the particular command we are testing. Third digit (optionally) tells the particular switch or group of switches we are testing. If you create files under t/ directory (i.e. here) that is not the top-level test script, never name the file to match the above pattern. The Makefile here considers all such files as the top-level test script and tries to run all of them. Care is especially needed if you are creating a common test library file, similar to test-lib.sh, because such a library file may not be suitable for standalone execution. Writing Tests ------------- The test script is written as a shell script. It should start with the standard "#!/bin/sh", and an assignment to variable 'test_description', like this: #!/bin/sh test_description='xxx test (option --frotz) This test registers the following structure in the cache and tries to run git-ls-files with option --frotz.' Source 'test-lib.sh' -------------------- After assigning test_description, the test script should source test-lib.sh like this: . ./test-lib.sh This test harness library does the following things: - If the script is invoked with command line argument --help (or -h), it shows the test_description and exits. - Creates an empty test directory with an empty .git/objects database and chdir(2) into it. This directory is 't/trash directory.$test_name_without_dotsh', with t/ subject to change by the --root option documented above, and a '.stress-' suffix appended by the --stress option. - Defines standard test helper functions for your scripts to use. These functions are designed to make all scripts behave consistently when command line arguments --verbose (or -v), --debug (or -d), and --immediate (or -i) is given. Recommended style ----------------- Here are some recommented styles when writing test case. - Keep test title the same line with test helper function itself. Take test_expect_success helper for example, write it like: test_expect_success 'test title' ' ... test body ... ' Instead of: test_expect_success \ 'test title' \ '... test body ...' - End the line with a single quote. - Indent the body of here-document, and use "<<-" instead of "<<" to strip leading TABs used for indentation: test_expect_success 'test something' ' cat >expect <<-\EOF && one two three EOF test_something > actual && test_cmp expect actual ' Instead of: test_expect_success 'test something' ' cat >expect <<\EOF && one two three EOF test_something > actual && test_cmp expect actual ' - Quote or escape the EOF delimiter that begins a here-document if there is no parameter and other expansion in it, to signal readers that they can skim it more casually: cmd <<-\EOF literal here-document text without any expansion EOF Do's & don'ts ------------- Here are a few examples of things you probably should and shouldn't do when writing tests. Here are the "do's:" - Put all code inside test_expect_success and other assertions. Even code that isn't a test per se, but merely some setup code should be inside a test assertion. - Chain your test assertions Write test code like this: git merge foo && git push bar && test ... Instead of: git merge hla git push gh test ... That way all of the commands in your tests will succeed or fail. If you must ignore the return value of something, consider using a helper function (e.g. use sane_unset instead of unset, in order to avoid unportable return value for unsetting a variable that was already unset), or prepending the command with test_might_fail or test_must_fail. - Check the test coverage for your tests. See the "Test coverage" below. Don't blindly follow test coverage metrics; if a new function you added doesn't have any coverage, then you're probably doing something wrong, but having 100% coverage doesn't necessarily mean that you tested everything. Tests that are likely to smoke out future regressions are better than tests that just inflate the coverage metrics. - When a test checks for an absolute path that a git command generated, construct the expected value using $(pwd) rather than $PWD, $TEST_DIRECTORY, or $TRASH_DIRECTORY. It makes a difference on Windows, where the shell (MSYS bash) mangles absolute path names. For details, see the commit message of 4114156ae9. - Remember that inside the