Start testing.md, and move to docs/

2 files changed, 89 insertions, 0 deletions

diff --git a/docs/releasing.md b/docs/releasing.md
new file mode 100644
index 0000000..6c553b3
--- /dev/null
+++ b/docs/releasing.md
@@ -0,0 +1,68 @@
+# Releasing
+
+This summarises what 'releasing' means for ybd - it's a work-in-progress. 
+
+### Past
+
+In the first months there were no 'releases' of ybd. The early goals of the
+project were to parse definitions as fast as possible, then build them, then
+deploy them.
+
+No automated tests were produced because
+- the project aimed (and still aims) to have as little code as possible
+- upstream wanted to show that reliable software can be written without tests
+- it was possible to test ybd on actual definitions
+
+### Present
+
+Currently the process is light-weight but ad-hoc:
+
+- ybd has no automated tests, so
+- changes are assumed to be tested by their creators
+- pull requests are typically merged with little or no upstream review/testing
+- breakages are discovered post-merge, typically by
+- a full run of ybd on ci.morph x86_64 and/or arm7lhf
+- broken changes to ybd are reverted (or sometimes fixed)
+- Pep8 is applied post-merge
+- releases are tagged periodically, usually after building ci.morph  
+
+There are some weaknesses in this:
+
+- many use-cases are not tested
+- breakages happen
+- a full build of ci.morph now takes 3 hours on a huge AWS machine
+- documentation can get out-of-step
+
+### Future
+
+Ultimately ybd releases should be as automated as possible, to make the
+process efficient (minimum work) and reliable (minimum breakage).
+
+The preferred approach for automation is expected to be one or more of
+- Concourse
+- GitLab CI
+- Travis
+
+
+### Some Test Cases
+
+This list aims to identify the minimum set that would give confidence that
+a new version ybd *works*.
+
+- verify installation still works, on
+  - Vagrant
+  - AWS
+  - Fedora
+  - Debian/Ubuntu
+- verify ybd still gets same cache-keys for a given set of definitions
+- build old releases of definitions, as far back as possible
+- build with empty, partial and full cache
+- check ctrl-c works, and that re-start works
+- verify reproducible components are reproduced
+
+Even to test this minimum list fully on real definitions will take a lot of
+server time, because actual system builds are big and heavy. To improve on
+this we'd benefit from establishing a reference definition set which is
+specifically for exercising ybd/definitions/spec.
+
+
diff --git a/docs/testing.md b/docs/testing.md
new file mode 100644
index 0000000..4b2ee75
--- /dev/null
+++ b/docs/testing.md
@@ -0,0 +1,21 @@
+check_cache_keys
+----------------
+Since ybd 15.44, the cache-key algorithm appears stable and can be
+applied to definitions as far back as baserock-14.40.
+
+So we can use the cache-keys as checksums to avoid regressions in the
+algorithm. This doesn't matter much for the cache-key algorithm itself,
+since it is versioned, but checking that we can get to these values also
+confirms that we can still parse the old definitions files themselves.
+
+baserock-14.40
+v0: ci.6788f6634d42f0a42682bd2208cb123f69a157b4a50ac40108d91029558ec623
+v1: ci.b9de86669ce182e60e3f9445e6394b478b67a2c73b4c0764491c158c5f2569e9
+
+baserock-14.46
+v0: ci.6357ea38e22bcbc3ebc399d36ed3bbfe03cab79079ab5b97096b7fc36e0e3351
+v1: ci.99203daccc211d634102f94cab7095c7dae31c6daadc7541380d53e1b3b2fe8e
+
+baserock-15.47.1
+v0: ci.a809e39d26fea31b2dc674e190f6367a10cedaffe644775a59a93719cc5d5290
+v1: ci.ce7f81b2e294c1893a08e4b18cf1d9a74f5e49bb58f7099ab55f656a05d887b4