From 3c0b924c9d031cee8f3f9da7bb2405bf65ef30bb Mon Sep 17 00:00:00 2001
From: Cyrill Gorcunov <gorcunov@gmail.com>
Date: Wed, 19 Dec 2018 18:57:05 +0300
Subject: test: nasm-t -- Add more details into README

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
---
 travis/README.md | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 69 insertions(+), 7 deletions(-)

diff --git a/travis/README.md b/travis/README.md
index d5aee054..d8f1c4dd 100644
--- a/travis/README.md
+++ b/travis/README.md
@@ -26,14 +26,50 @@ A typical test case processed by the following steps:
 
 Use `nasm-t.py -h` command to get the detailed description of every option.
 
+### Test unit structure
+Each test consists at least of three files:
+
+ - a test descriptor in with `*.json` extension;
+ - a source file to compile;
+ - a target file to compare result with, it is assumed to have
+   the same name as output generated during the pass file but with `*.t`
+   extension; thus if a test generates `*.bin` file the appropriate target
+   should have `*.bin.t` name.
+
+Running tests
+-------------
+To run all currently available tests simply type the following
+
+```console
+python3 travis/nasm-t.py run
+```
+
+By default the `nasm-t.py` scans `test` subdirectory for `*.json` files and
+consider each as a test descriptor. Then every test is executed sequentially.
+If the descriptor can not be parsed it silently ignored.
+
+To run a particular test provide the test name, for example
+
+```console
+python3 travis/nasm-t.py list
+...
+./travis/test/utf                Test __utf__ helpers
+./travis/test/utf                Test errors in __utf__ helpers
+...
+python3 travis/nasm-t.py run -t ./travis/test/utf
+```
+
+Test name duplicates in the listing above means that the descriptor
+carries several tests with same name but different options.
+
 Test descriptor file
 --------------------
 A descriptor file should provide enough information how to run the NASM
 itself and which output files or streams to compare with predefined ones.
-We use `JSON` format with the following fields:
+We use *JSON* format with the following fields:
 
  - `description`: a short description of a test which is shown to
-   a user when tests are listed;
+   a user when tests are being listed;
  - `id`: descriptor internal name to use with `ref` field;
  - `ref`: a reference to `id` from where settings should be
    copied, it is convenient when say only `option` is different
@@ -49,14 +85,14 @@ We use `JSON` format with the following fields:
     - `stderr`: a file containing *stderr* stream output to check;
     - `stdout`: a file containing *stdout* stream output to check;
     - `output`: a file containing compiled result to check, in other
-      words it is a name passed as `-o` option to the compiler.
-
-Examples
---------
+      words it is a name passed as `-o` option to the compiler;
+ - `error`: an error handler, can be either *over* to ignore any
+   error happened, or *expected* to make sure the test is failing.
 
+### Examples
 A simple test where no additional options are used, simply compile
 `absolute.asm` file with `bin` format for output, then compare
-produced `absolute.bin` file with precompiled `absolute.bin.dest`.
+produced `absolute.bin` file with precompiled `absolute.bin.t`.
 
 ```json
 {
@@ -69,6 +105,10 @@ produced `absolute.bin` file with precompiled `absolute.bin.dest`.
 }
 ```
 
+Note the `output` target is named as *absolute.bin* where *absolute.bin.t*
+should be already precompiled (we will talk about it in `update` action)
+and present on disk.
+
 A slightly complex example: compile one source file with different optimization
 options and all results must be the same. To not write three descriptors
 we assign `id` to the first one and use `ref` term to copy settings.
@@ -102,3 +142,25 @@ warnings to compare.
 	}
 ]
 ```
+
+Updating tests
+--------------
+If during development some of the targets are expected to change
+the tests will start to fail so the should be updated. Thus new
+precompiled results will be treated as templates to compare with.
+
+To update all tests in one pass run
+
+```console
+python3 travis/nasm-t.py update
+...
+=== Updating ./travis/test/xcrypt ===
+	Processing ./travis/test/xcrypt
+	Executing ./nasm -f bin -o ./travis/test/xcrypt.bin ./travis/test/xcrypt.asm
+	Moving ./travis/test/xcrypt.bin to ./travis/test/xcrypt.bin.t
+=== Test ./travis/test/xcrypt UPDATED ===
+...
+```
+
+and commit the results. To update a particular test provide its name
+with `-t` option.
-- 
cgit v1.2.1