Draft of new test appraoches for parsing and splitting

1 files changed, 40 insertions, 6 deletions

diff --git a/docs/testing.md b/docs/testing.md
index 7317cb7..3c9254b 100644
--- a/docs/testing.md
+++ b/docs/testing.md
@@ -1,3 +1,11 @@
+check_parsing
+-------------
+YBD aims to normalise definitions into a more understandable representation.
+With the addition of 'parse-only' mode, ybd can dump this representation as
+json, and we can confirm there is no unexpected data.
+
+FIXME: identify at least one valid test-case here.
+
 check_cache_keys
 ----------------
 Since ybd 15.44, the cache-key algorithm appears stable and can be
@@ -10,14 +18,40 @@ confirms that we can still parse the old definitions files themselves.
 
 ```
     baserock-14.40
-    v0: ci.6788f6634d42f0a42682bd2208cb123f69a157b4a50ac40108d91029558ec623
-    v1: ci.b9de86669ce182e60e3f9445e6394b478b67a2c73b4c0764491c158c5f2569e9
+    0: ci.6788f6634d42f0a42682bd2208cb123f69a157b4a50ac40108d91029558ec623
+    1: ci.b9de86669ce182e60e3f9445e6394b478b67a2c73b4c0764491c158c5f2569e9
 
     baserock-14.46
-    v0: ci.6357ea38e22bcbc3ebc399d36ed3bbfe03cab79079ab5b97096b7fc36e0e3351
-    v1: ci.99203daccc211d634102f94cab7095c7dae31c6daadc7541380d53e1b3b2fe8e
+    0: ci.6357ea38e22bcbc3ebc399d36ed3bbfe03cab79079ab5b97096b7fc36e0e3351
+    1: ci.99203daccc211d634102f94cab7095c7dae31c6daadc7541380d53e1b3b2fe8e
 
     baserock-15.47.1
-    v0: ci.a809e39d26fea31b2dc674e190f6367a10cedaffe644775a59a93719cc5d5290
-    v1: ci.ce7f81b2e294c1893a08e4b18cf1d9a74f5e49bb58f7099ab55f656a05d887b4
+    0: ci.a809e39d26fea31b2dc674e190f6367a10cedaffe644775a59a93719cc5d5290
+    1: ci.ce7f81b2e294c1893a08e4b18cf1d9a74f5e49bb58f7099ab55f656a05d887b4
+```
+
+check splits
+------------
+We reduce the size of systems artifacts by installing a only subset of the
+outputs from the builds during the assembly process. The code for this
+is mainly in splitting.py, and involves applying regex expressions to the
+artifacts, based on the definitions and default split-rules, during the
+'install-stratum-artifacts' step.
+
+For a given checkout of definitions, the generated files in a system should
+be the same for all future versions of ybd, so we can checksum a known-good
+list of generated files, and compare new runs against it as a regression test.
+
+The best candidate is minimal-system, since this expressly makes use of a lot
+of splitting. The proposed test case is from Richard Dale's original commit
+of the split functionality...
+
+```
+    FIXME: when Richard's splitting was introduced, it seems YBD could not
+           build minimal system!!!
+    definitions version: baserock-16.13
+    artifact-version: 3
+    ybd version: 16.01-29-gb7c9206 (16.01 + 29 commits)
+    command: cat baserock/* | grep \/ | sort  | md5sum
+    expected output: e5fb0693b618213e3d9bf43ea6877a43
 ```