diff options
author | David Percy <david.percy@mongodb.com> | 2022-07-12 20:24:57 +0000 |
---|---|---|
committer | Evergreen Agent <no-reply@evergreen.mongodb.com> | 2022-08-17 15:37:04 +0000 |
commit | b902366ad23713ed7493e121f0218fead287bc6d (patch) | |
tree | 762de8a4a020d82ff3cf43faf7a35ba736e37a30 /docs | |
parent | aca57c0414496efbfaa914343e3b599366f15a14 (diff) | |
download | mongo-b902366ad23713ed7493e121f0218fead287bc6d.tar.gz |
SERVER-67415 Create golden-data JS suites
Adds two new suites: query_golden_classic and query_golden_cqf, which
use the golden-data test framework (docs/golden_data_test_framework.md).
These suites diff the expected and actual output instead of using
assert.eq and similar functions.
This change includes some refactoring of the golden-test classes to avoid
pulling the unittest library into the mongo shell.
Diffstat (limited to 'docs')
-rw-r--r-- | docs/golden_data_test_framework.md | 21 |
1 files changed, 11 insertions, 10 deletions
diff --git a/docs/golden_data_test_framework.md b/docs/golden_data_test_framework.md index e01c6a502e8..fd99f0c8527 100644 --- a/docs/golden_data_test_framework.md +++ b/docs/golden_data_test_framework.md @@ -1,6 +1,6 @@ # Overview Golden Data test framework provides ability to run and manage tests that produce an output which is -verified by comparing it to the checked-in, known valid output. Any differences result in rest +verified by comparing it to the checked-in, known valid output. Any differences result in test failure and either the code or expected output has to be updated. Golden Data tests excel at bulk diffing of failed test outputs and bulk accepting of new test @@ -160,7 +160,7 @@ buildscripts/golden_test.py requires a one-time workstation setup. Note: this setup is only required to use buildscripts/golden_test.py itself. It is NOT required to just run the Golden Data tests when not using buildscripts/golden_test.py. -1. Create a yaml config file, as described by golden_test_options.idl (link TBD). +1. Create a yaml config file, as described by [Appendix - Config file reference](#appendix---config-file-reference). 2. Set GOLDEN_TEST_CONFIG_PATH environment variable to config file location, so that is available when running tests and when running buildscripts/golden_test.py tool. @@ -198,19 +198,19 @@ alternatively modify /etc/environment or other configuration if needed by Debugg ### List all available test outputs ```bash -mongo > buildscripts/golden_test.py list +$> buildscripts/golden_test.py list ``` ### Diff test results from most recent test run: ```bash -mongo > buildscripts/golden_test.py diff +$> buildscripts/golden_test.py diff ``` This will run the diffCmd that was specified in the config file ### Diff test results from most recent test run: ```bash -mongo > buildscripts/golden_test.py accept +$> buildscripts/golden_test.py accept ``` This will copy all actual test outputs from that test run to the source repo and new expected outputs. @@ -219,17 +219,17 @@ outputs. ### Get paths from most recent test run (to be used by custom tools) Get expected and actual output paths for most recent test run: ```bash -mongo > buildscripts/golden_test.py get +$> buildscripts/golden_test.py get ``` Get expected and actual output paths for most most recent test run: ```bash -mongo > buildscripts/golden_test.py get_root +$> buildscripts/golden_test.py get_root ``` Get all available commands and options: ```bash -mongo > buildscripts/golden_test.py --help +$> buildscripts/golden_test.py --help ``` # How to diff test results from a non-workstation test run @@ -281,9 +281,10 @@ diffCmd: Shell command to diff a single golden test run output. {{expected}} and {{actual}} variables should be used and will be replaced with expected and actual output folder paths respectively. - This property is not used during test execution, it is only used by post processing tools, - like golden_test.py + This property is not used to decide whether the test passes or fails; it is only used to + display differences once we've decided that a test failed. examples: diff -ruN --unidirectional-new-file --color=always "{{expected}}" "{{actual}}" + git diff --no-index "{{expected}}" "{{actual}}" ``` |