path: root/docs/golden_data_test_framework.md

# Overview
Golden Data test framework provides ability to run and manage tests that produce an output which is 
verified by comparing it to the checked-in, known valid output. Any differences result in test 
failure and either the code or expected output has to be updated.

Golden Data tests excel at bulk diffing of failed test outputs and bulk accepting of new test 
outputs. 

# When to use Golden Data tests?
* Code under test produces a deterministic output: That way tests can consistently succeed or fail.
* Incremental changes to code under test or test fixture result in incremental changes to the 
output.
* As an alternative to ASSERT for large output comparison: Serves the same purpose, but provides
 tools for diffing/updating.
* The outputs can't be objectively verified (e.g. by verifying well known properties). Examples:
  * Verifying if sorting works, can be done by verifying that output is sorted. SHOULD NOT use 
  Golden Data tests.
  * Verifying that pretty printing works, MAY use Golden Data tests to verify the output, as there
   might not be well known properties or those properties can easily change.
* As stability/versioning/regression testing. Golden Data tests by storing recorded outputs, are 
good candidate for preserving behavior of legacy versions or detecting undesired changes in 
behavior, even in cases when new behavior meets other correctness criteria.

# Best practices for working with Golden Data tests
* Tests MUST produce text output that is diffable can be inspected in the pull request.

* Tests MUST produce an output that is deterministic and repeatable. Including running on different
platforms. Same as with ASSERT_EQ.
* Tests SHOULD produce an output that changes incrementally in response to the incremental test or
code changes. 

* Multiple test variations MAY be bundled into a single test. Recommended when testing same feature 
with different inputs. This helps reviewing the outputs by grouping similar tests together, and also 
reduces the number of output files.

* Changes to test fixture or test code that affect non-trivial amount test outputs MUST BE done in 
separate pull request from production code changes:
  * Pull request for test code only changes can be easily reviewed, even if large number of test 
  outputs are modified. While such changes can still introduce merge conflicts, they don't introduce
   risk of regression (if outputs were valid
  * Pull requests with mixed production 

* Tests in the same suite SHOULD share the fixtures when appropriate. This reduces cost of adding 
new tests to the suite. Changes to the fixture may only affect expected outputs from that fixtures, 
and those output can be updated in bulk.

* Tests in different suites SHOULD NOT reuse/share fixtures. Changes to the fixture can affect large
 number of expected outputs. 
  There are exceptions to that rule, and tests in different suites MAY reuse/share fixtures if:
  * Test fixture is considered stable and changes rarely.
  * Tests suites are related, either by sharing tests, or testing similar components.
  * Setup/teardown costs are excessive, and sharing the same instance of a fixture for performance 
  reasons can't be avoided.

* Tests SHOULD print both inputs and outputs of the tested code. This makes it easy for reviewers to 
verify of the expected outputs are indeed correct by having both input and output next to each 
other. 
Otherwise finding the input used to produce the new output may not be practical, and might not even 
be included in the diff.

* When resolving merge conflicts on the expected output files, one of the approaches below SHOULD be 
used:
  * "Accept theirs", rerun the tests and verify new outputs. This doesn't require knowledge of 
  production/test code changes in "theirs" branch, but requires re-review and re-acceptance of c
  hanges done by local branch.
  * "Accept yours", rerun the tests and verify the new outputs. This approach requires knowledge of 
  production/test code changes in "theirs" branch. However, if such changes resulted in 
  straightforward and repetitive output changes, like due to printing code change or fixture change,
   it may be easier to verify than reinspecting local changes.

* Expected test outputs SHOULD be reused across tightly-coupled test suites. The suites are 
tightly-coupled if:
  * Share the same tests, inputs and fixtures.
  * Test similar scenarios.
  * Test different code paths, but changes to one of the code path is expected to be accompanied by 
  changes to the other code paths as well.
 
  Tests SHOULD use different test files, for legitimate and expected output differences between 
  those suites.

  Examples:
  * Functional tests, integration tests and unit tests that test the same behavior in different 
  environments.
  * Versioned tests, where expected behavior is the same for majority of test inputs/scenarios. 

* AVOID manually modifying expected output files. Those files are considered to be auto generated. 
Instead, run the tests and then copy the generated output as a new expected output file. See "How to
 diff and accept new test outputs" section for instructions.


# How to use write Golden Data tests?
Each golden data test should produce a text output that will be later verified. The output format 
must be text, but otherwise test author can choose a most appropriate output format (text, json, 
bson, yaml or mixed). If a test consists of multiple variations each variation should be clearly 
separated from each other. 

Note: Test output is usually only written. It is ok to focus on just writing serialization/printing 
code without a need to provide deserialization/parsing code. 

When actual test output is different from expected output, test framework will fail the test, log
 both outputs and also create following files, that can be inspected later:
* <output_path>/actual/<test_path> - with actual test output
* <output_path>/expected/<test_path> - with expected test output

## CPP tests
`::mongo::unittest::GoldenTestConfig` - Provides a way to configure test suite(s). Defines where the
 expected output files are located in the source repo.

`::mongo::unittest::GoldenTestContext` - Provides an output stream where tests should write their 
outputs. Verifies the output with the expected output that is in the source repo

See: [golden_test.h](../src/mongo/unittest/golden_test.h)

**Example:**
```c++
#include "mongo/unittest/golden_test.h"

GoldenTestConfig myConfig("src/mongo/my_expected_output");
TEST(MySuite, MyTest) {
    GoldenTestContext ctx(myConfig);
    ctx.outStream() << "print something here" << std::endl;
    ctx.outStream() << "print something else" << std::endl;
}

void runVariation(GoldenTestContext& ctx, const std::string& variationName, T input) {
    ctx.outStream() << "VARIATION " << variationName << std::endl;
    ctx.outStream() << "input: " << input << std::endl;
    ctx.outStream() << "output: " << runCodeUnderTest(input) << std::endl;
    ctx.outStream() << std::endl;
}

TEST_F(MySuiteFixture, MyFeatureATest) {
    GoldenTestContext ctx(myConfig);
    runMyVariation(ctx, "variation 1", "some input testing A #1")
    runMyVariation(ctx, "variation 2", "some input testing A #2")
    runMyVariation(ctx, "variation 3", "some input testing A #3")
}

TEST_F(MySuiteFixture, MyFeatureBTest) {
    GoldenTestContext ctx(myConfig);
    runMyVariation(ctx, "variation 1", "some input testing B #1")
    runMyVariation(ctx, "variation 2", "some input testing B #2")
    runMyVariation(ctx, "variation 3", "some input testing B #3")
    runMyVariation(ctx, "variation 4", "some input testing B #4")
}
```

Also see self-test: 
[golden_test_test.cpp](../src/mongo/unittest/golden_test_test.cpp)

# How to diff and accept new test outputs on a workstation

Use buildscripts/golden_test.py command line tool to manage the test outputs. This includes:
* diffing all output differences of all tests in a given test run output. 
* accepting all output differences of all tests in a given test run output. 

## Setup
buildscripts/golden_test.py requires a one-time workstation setup.

Note: this setup is only required to use buildscripts/golden_test.py itself. It is NOT required to
just run the Golden Data tests when not using buildscripts/golden_test.py.

1. Create a yaml config file, as described by [Appendix - Config file reference](#appendix---config-file-reference).
2. Set GOLDEN_TEST_CONFIG_PATH environment variable to config file location, so that is available 
when running tests and when running buildscripts/golden_test.py tool.

### Automatic Setup

Use buildscripts/golden_test.py builtin setup to initialize default config for your current platform.

**Instructions for Linux**

Run buildscripts/golden_test.py setup utility
```bash
buildscripts/golden_test.py setup
```

**Instructions for Windows**

Run buildscripts/golden_test.py setup utility.
You may be asked for a password, when not running in "Run as administrator" shell.
```cmd
c:\python\Python37\python.exe buildscripts/golden_test.py setup
```

### Manual Setup (Default config)

This is the same config as that would be setup by the [Automatic Setup](#automatic-setup)

This config uses a unique subfolder folder for each test run. (default)
   * Allows diffing each test run separately. 
   * Works with multiple source repos.

**Instructions for Linux/macOS:**

This config uses a unique subfolder folder for each test run. (default)
   * Allows diffing each test run separately. 
   * Works with multiple source repos.

Create ~/.golden_test_config.yml with following contents:
```yaml
outputRootPattern: /var/tmp/test_output/out-%%%%-%%%%-%%%%-%%%%
diffCmd: git diff --no-index "{{expected}}" "{{actual}}"
```

Update .bashrc, .zshrc
```bash
export GOLDEN_TEST_CONFIG_PATH=~/.golden_test_config.yml
```
alternatively modify /etc/environment or other configuration if needed by Debugger/IDE etc.

**Instructions for Windows:**

Create %LocalAppData%\.golden_test_config.yml with the following contents:
```yaml
outputRootPattern: 'C:\Users\Administrator\AppData\Local\Temp\test_output\out-%%%%-%%%%-%%%%-%%%%'
diffCmd: 'git diff --no-index "{{expected}}" "{{actual}}"'
```

Add GOLDEN_TEST_CONFIG_PATH=~/.golden_test_config.yml environment variable:
```cmd
runas /profile /user:administrator "setx GOLDEN_TEST_CONFIG_PATH %LocalAppData%\.golden_test_config.yml"
```

## Usage

### List all available test outputs
```bash
$> buildscripts/golden_test.py list
```

### Diff test results from most recent test run:

```bash
$> buildscripts/golden_test.py diff
```
This will run the diffCmd that was specified in the config file

### Diff test results from most recent test run:
```bash
$> buildscripts/golden_test.py accept
```
This will copy all actual test outputs from that test run to the source repo and new expected 
outputs.


### Get paths from most recent test run (to be used by custom tools)
Get expected and actual output paths for most recent test run:
```bash
$> buildscripts/golden_test.py get
```

Get expected and actual output paths for most most recent test run:
```bash
$> buildscripts/golden_test.py get_root
```

Get all available commands and options:
```bash
$> buildscripts/golden_test.py --help
```

# How to diff test results from a non-workstation test run

## Bulk folder diff the results:
1. Parse the test log to find the root output locations where expected and actual output files were 
written.
2. Then compare the folders to see the differences for tests that failed.

**Example: (linux/macOS)**

```bash
# Show the test run expected and actual folders:
$> cat test.log | grep "^{" | jq -s -c -r '.[] | select(.id == 6273501 ) | .attr.expectedOutputRoot + " " +.attr.actualOutputRoot ' | sort | uniq
# Run the recursive diff
$> diff -ruN --unidirectional-new-file --color=always <expected_root> <actual_root>
```

## Find the outputs of tests that failed.
Parse logs and find the the expected and actual outputs for each failed test.

**Example: (linux/macOS)**
```bash
# Find all expected and actual outputs of tests that have failed
$> cat test.log | grep "^{" | jq -s '.[] | select(.id == 6273501 ) | .attr.testPath,.attr.expectedOutput,.attr.actualOutput'
```

# Appendix - Config file reference

Golden Data test config file is a YAML file specified as:
```yaml
outputRootPattern: 
    type: String
    optional: true
    description:
         Root path patten that will be used to write expected and actual test outputs for all tests 
         in the test run.
         If not specified a temporary folder location will be used.
         Path pattern string may use '%' characters in the last part of the path. '%' characters in
         the last part of the path will be replaced with random lowercase hexadecimal digits.
    examples:
         /var/tmp/test_output/out-%%%%-%%%%-%%%%-%%%%
         /var/tmp/test_output

diffCmd: 
    type: String
    optional: true
    description:  
        Shell command to diff a single golden test run output.
        {{expected}} and {{actual}} variables should be used and will be replaced  with expected and
        actual output folder paths respectively.
        This property is not used to decide whether the test passes or fails; it is only used to
        display differences once we've decided that a test failed.
    examples:
         git diff --no-index "{{expected}}" "{{actual}}"
         diff -ruN --unidirectional-new-file --color=always "{{expected}}" "{{actual}}"
```