diff options
| author | Lars Wirzenius <lars.wirzenius@codethink.co.uk> | 2013-07-23 16:17:02 +0000 |
|---|---|---|
| committer | Lars Wirzenius <lars.wirzenius@codethink.co.uk> | 2013-07-23 16:17:02 +0000 |
| commit | 27ecfc8d5a2dd5423993d830e3034cce8f09ee9f (patch) | |
| tree | 44303bc90008dd0770f3150e7ae32b08745f2bbb | |
| parent | ae917eb8bfae45af8b95285a561b8249f025cd9e (diff) | |
| parent | e56ae9f1a75f44a702d1965bc543e84b43db2fff (diff) | |
| download | cmdtest-27ecfc8d5a2dd5423993d830e3034cce8f09ee9f.tar.gz | |
Merge branch 'master' into liw/update-cmdtest
Get current upstream into Baserock. Most importantly, this includes
the new yarn tool.
33 files changed, 1621 insertions, 12 deletions
@@ -3,7 +3,57 @@ NEWS for cmdtest This file summarizes changes between releases of cmdtest. -Version 0.6, released UNRELEASED +Version 0.X, released UNRELEASED +-------------------------------- + +* Yarn now warns if an input file has no code blocks. +* There is no a yarn `--shell-library` option for the user to use, which + includes a shell library when running any IMPLEMENTS section. +* FINALLY always worked in yarn, but has now been added to the manual + page as well. +* The keyword ASSUMING has been added to yarn. +* New yarn option `--run` allows running selected tests only. +* New yarn option `--snapshot` makes snapshots of the test working + directory after each step in a scenario. Combined with the, also + new, option `--tempdir` this makes debugging failed tests easier. +* New yarn option `--verbose` (turned on automatically if there is not + tty available, e.g., when running from cron), turns off ttystatus + progress bar and produces a "wall of text" style output instead. + +Bug fixes: + +* Yarn now handles multiple input files correctly. Reported by Daniel + Silverstone; fix based on his patch, but rewritten. + +Version 0.8.3, released 2013-06-21 +-------------------------------- + +* Bug fix: properly install yarnlib. In other news, I hate distutils. + +Version 0.8.2, released 2013-06-21 +-------------------------------- + +* Bug fix: install the yarnlib library as well. This is embarrassing. + +Version 0.8.1, released 2013-06-20 +-------------------------------- + +* Bug fix: install the yarn binary in the package. + +Version 0.8, released 2013-06-19 +-------------------------------- + +* Switch terminology to "scenario testing" from "story testing". Thanks + to Rob Kendrick for the suggestion. Doing a quick release so the + old terminology doesn't have time to get any use. + +Version 0.7, released 2013-06-15 +-------------------------------- + +* Added the new tool `yarn`, for doing story testing. It is still fresh + and raw, but I want to make it available to get feedback. + +Version 0.6, released 2013-03-14 -------------------------------- * Fixed cmdtest to diff outputs correctly. Reported by Kevin Lee. @@ -1,6 +1,13 @@ README for cmdtest ================== +This project consists of two programs: the original `cmdtest`, +and the newer `yarn`. Both are black box testing tools for Unix +command line tools. + +cmdtest +------- + `cmdtest` black box tests Unix command line tools. Given some test scripts, their inputs, and expected outputs, it verifies that the command line produces the expected output. @@ -9,10 +16,27 @@ If not, it reports problems, and shows the differences. See the manual page for details on how to use the program. +yarn +---- + +`yarn` also black box tests Unix command line tools, but takes +a different approach, where the emphasis is on verifying that the +tools works correctly in a sequence of operations, or +what we call a "test scenario". `yarn` is inspired [BDD][BDD], +behavior-driven development, and some of the implementations made +by the Ruby community. + +See README.yarn for more details. + +`yarn` has been designed with Daniel Silverstone. + +[BDD]: https://en.wikipedia.org/wiki/Behavior-driven_development + + Legalese -------- -Copyright 2011 Lars Wirzenius +Copyright 2011-2013 Lars Wirzenius This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by diff --git a/README.yarn b/README.yarn new file mode 100644 index 0000000..76d57b5 --- /dev/null +++ b/README.yarn @@ -0,0 +1,183 @@ +README for yarn, a scenario testing tool +======================================== + +Introduction +------------ + +`yarn` is a scenario testing tool: you write a scenario describing how a +user uses your software and what should happen, and express, using +very lightweight syntax, the scenario in such a way that it can be tested +automatically. The scenario has a simple, but strict structure: + + SCENARIO name of scenario + GIVEN some setup for the test + WHEN thing that is to be tested happens + THEN the post-conditions must be true + +As an example, consider a very short test scenario for verifying that +a backup program works, at least for one simple case. + + SCENARIO basic backup and restore + GIVEN some live data in a directory + AND an empty backup repository + WHEN a backup is made + THEN the data can be restored + +(Note the addition of AND: you can have multiple GIVEN, WHEN, and +THEN statements. The AND keyword makes the text be more readable.) + +Scenarios are meant to be written in mostly human readable language. +However, they are not free form text. In addition to the GIVEN/WHEN/THEN +structure, the text for each of the steps needs a computer-executable +implementation. This is done by using IMPLEMENTS. The backup scenario +from above might be implemented as follows: + + IMPLEMENTS GIVEN some live data in a directory + rm -rf "$DATADIR/data" + mkdir "$DATADIR/data" + echo foo > "$DATADIR/data/foo" + + IMPLEMENTS GIVEN an empty backup repository + rm -rf "$DATADIR/repo" + mkdir "$DATADIR/repo" + + IMPLEMENTS WHEN a backup is made + backup-program -r "$DATADIR/repo" "$DATADIR/data" + + IMPLEMENTS THEN the data can be restored + mkdir "$DATADIR/restored" + restore-program -r "$DATADIR/repo" "$DATADIR/restored" + diff -rq "$DATADIR/data" "$DATADIR/restored" + +Each "IMPLEMENT GIVEN" (or WHEN, THEN) is followed by a regular +expression on the same line, and then a shell script that gets executed +to implement any step that matches the regular expression. The +implementation can extract data from the match as well: for example, +the regular expression might allow a file size to be specified. + +The above example seems a bit silly, of course: why go to the effort +to obfuscate the various steps? The answer is that the various steps, +implemented using IMPLEMENTS, can be combined in many ways, to test +different aspects of the program being tested. In effect, the IMPLEMENTS +sections provide a vocabulary which the scenario writer can use to +express a variety of usefully different scenarios, which together +test all the aspects of the software that need to be tested. + +Moreover, by making the step descriptions be human language +text, matched by regular expressions, most of the test can +hopefully be written, and understood, by non-programmers. Someone +who understands what a program should do, could write tests +to verify its behaviour. The implementations of the various +steps need to be implemented by a programmer, but given a +well-designed set of steps, with enough flexibility in their +implementation, that quite a good test suite can be written. + +Test language specification +--------------------------- + +A test document is written in [Markdown][markdown], with block +quoted code blocks being interpreted specially. Each block +must follow the syntax defined here. + +* Every step in a scenario is one line, and starts with a keyword. + +* Each implementation (IMPLEMENTS) starts as a new block, and + continues until there is a block that starts with another + keyword. + +The following keywords are defined. + +* **SCENARIO** starts a new scenario. The rest of the line is the name of + the scenario. The name is used for documentation and reporting + purposes only and has no semantic meaning. SCENARIO MUST be the + first keyword in a scenario, with the exception of IMPLEMENTS. + The set of documents passed in a test run may define any number of + scenarios between them, but there must be at least one or it is a + test failure. The IMPLEMENTS sections are shared between the + documents and scenarios. + +* **ASSUMING** defines a condition for the scenario. The rest of the + line is "matched text", which gets implemented by an + IMPLEMENTS section. If the code executed by the implementation + fails, the scenario is skipped. + +* **GIVEN** prepares the world for the test to run. If + the implementation fails, the scenario fails. + +* **WHEN** makes the change to the world that is to be tested. + If the code fails, the scenario fails. + +* **THEN** verifies that the changes made by the GIVEN steps + did the right thing. If the code fails, the scenario fails. + +* **FINALLY** specifies how to clean up after a scenario. If the code + fails, the scenario fails. All FINALLY blocks get run either when + encountered in the scenario flow, or at the end of the scenario, + regardless of whether the scenario is failing or not. + +* **AND** acts as ASSUMING, GIVEN, WHEN, THEN, or FINALLY: whichever + was used last. It must not be used unless the previous step was + one of those, or another AND. + +* **IMPLEMENTS** is followed by one of ASSUMING, GIVEN, WHEN, or THEN, + and a PCRE regular expression, all on one line, and then further + lines of shell commands until the end of the block quoted code + block. Markdown is unclear whether an empty line (no characters, + not even whitespace) between two block quoted code blocks starts a + new one or not, so we resolve the ambiguity by specifiying that a + code block directly following a code block is a continuation unless + it starts with one of the scenario testing keywords. + + The shell commands get parenthesised parts of the match of the + regular expression as environment variables (`$MATCH_1` etc). For + example, if the regexp is "a (\d+) byte file", then `$MATCH_1` gets + set to the number matched by `\d+`. + + The test runner creates a temporary directory, whose name is + given to the shell code in the `DATADIR` environment variable. + + The shell commands get invoked with `/bin/sh -eu`, and need to + be written accordingly. Be careful about commands that return a + non-zero exit code. There will eventually be a library of shell + functions supplied which allow handling the testing of non-zero + exit codes cleanly. In addition functions for handling stdout and + stderr will be provided. + + The code block of an IMPLEMENTS block fails if the shell + invocation exits with a non-zero exit code. Output to stderr is + not an indication of failure. Any output to stdout or stderr may + or may not be shown to the user. + +Semantics: + +* The name of each scenario (given with SCENARIO) must be unique. +* All names of scenarios and steps will be normalised before use + (whitespace collapse, leading and trailing whitespace +* Every ASSUMING, GIVEN, WHEN, THEN, FINALLY must be matched by + exactly one IMPLEMENTS. The test runner checks this before running + any code. +* Every IMPLEMENTS may match any number of ASSUMING, GIVEN, WHEN, + THEN, or FINALLY. The test runner may warn if an IMPLEMENTS is unused. +* If ASSUMING fails, that scenario is skipped, and any FINALLY steps + are not run. + +See also +-------- + +Wikipedia has an article on [Behaviour Driven Development][BDD], +which can provide background and further explanation to what this +tools tries to do. + +[BDD]: https://en.wikipedia.org/wiki/Behavior-driven_development +[Markdown]: http://daringfireball.net/projects/markdown/ + +TODO +---- + +* Add DEFINING, PRODUCING, if they turn out to be useful. +* Need something like ASSUMING, except fail the scenario if the + pre-condition is not true. Useful for testing that you can ssh + to localhost when flinging, for example. + **DJAS**: We think this might be 'REQUIRING' and it still does + not run the FINALLY group. + diff --git a/cmdtestlib.py b/cmdtestlib.py index 0b1684b..e6ca6bf 100644 --- a/cmdtestlib.py +++ b/cmdtestlib.py @@ -14,7 +14,7 @@ # along with this program. If not, see <http://www.gnu.org/licenses/>. -__version__ = '0.5' +__version__ = '0.8.3' import os diff --git a/debian/changelog b/debian/changelog index 353dc8a..46eb065 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,39 @@ +cmdtest (0.8.3-1) unstable; urgency=low + + * New upstream. + + -- Lars Wirzenius <liw@liw.fi> Fri, 21 Jun 2013 22:26:37 +0100 + +cmdtest (0.8.2-1) unstable; urgency=low + + * New upstream release. + + -- Lars Wirzenius <liw@liw.fi> Fri, 21 Jun 2013 20:33:43 +0100 + +cmdtest (0.8.1-1) unstable; urgency=low + + * New upstream release. + + -- Lars Wirzenius <liw@liw.fi> Thu, 20 Jun 2013 21:36:05 +0100 + +cmdtest (0.8-1) unstable; urgency=low + + * New upstream version. + + -- Lars Wirzenius <liw@liw.fi> Wed, 19 Jun 2013 21:00:17 +0100 + +cmdtest (0.7-1) unstable; urgency=low + + * New upstream version. + + -- Lars Wirzenius <liw@liw.fi> Sat, 15 Jun 2013 12:57:47 +0100 + +cmdtest (0.6-1) unstable; urgency=low + + * New upstream release. + + -- Lars Wirzenius <liw@liw.fi> Thu, 14 Mar 2013 08:22:46 +0000 + cmdtest (0.5-1) unstable; urgency=low * New upstream release. diff --git a/debian/control b/debian/control index 71bb9a2..5944b5d 100644 --- a/debian/control +++ b/debian/control @@ -4,13 +4,13 @@ Section: python Priority: optional Standards-Version: 3.9.3 Build-Depends: debhelper (>= 7.3.8), python-all (>= 2.6.6-3~), - python-cliapp, python-ttystatus + python-cliapp, python-ttystatus, python-markdown X-Python-Version: >= 2.6 Package: cmdtest Architecture: all Depends: ${python:Depends}, ${misc:Depends}, python (>= 2.6), python-cliapp, - python-ttystatus + python-ttystatus, python-markdown Description: blackbox testing of Unix command line programs cmdtest black box tests Unix command line tools. Roughly, it is given a a script, its input files, and its expected output files. cmdtest runs @@ -18,4 +18,6 @@ Description: blackbox testing of Unix command line programs . cmdtest is aimed specifically at testing non-interactive Unix command line programs, and tries to make that as easy as possible. + . + Also included is a "scenario testing" tool, yarn. Homepage: http://liw.fi/cmdtest/ diff --git a/run-yarn b/run-yarn new file mode 100755 index 0000000..596996f --- /dev/null +++ b/run-yarn @@ -0,0 +1,5 @@ +#!/bin/sh + +set -eu + +./yarn --no-default-config --quiet "$@" @@ -26,13 +26,27 @@ import subprocess import cmdtestlib +try: + import markdown +except ImportError: + markdown_version = None +else: + if (hasattr(markdown, 'extensions') and + hasattr(markdown.extensions, 'Extension')): + markdown_version = True + else: + markdown_version = False + class GenerateManpage(build): def run(self): build.run(self) print 'building manpages' - for x in ['cmdtest']: + cmds = ['cmdtest'] + if markdown_version: + cmds.append('yarn') + for x in cmds: with open('%s.1' % x, 'w') as f: subprocess.check_call(['python', x, '--generate-manpage=%s.1.in' % x, @@ -51,17 +65,20 @@ class CleanMore(clean): class Check(Command): user_options = [] - + def initialize_options(self): pass - + def finalize_options(self): pass def run(self): - subprocess.check_call(['python', '-m', 'CoverageTestRunner']) - os.remove('.coverage') - + if markdown_version: + subprocess.check_call( + ['python', '-m', 'CoverageTestRunner', + '--ignore-missing-from', 'without-tests']) + os.remove('.coverage') + subprocess.check_call(['./cmdtest', 'echo-tests']) subprocess.check_call(['./cmdtest', 'sort-tests']) @@ -72,6 +89,9 @@ class Check(Command): else: raise Exception('fail-tests did not fail, which is a surprise') + if markdown_version: + subprocess.check_call(['./cmdtest', 'yarn.tests']) + setup(name='cmdtest', version=cmdtestlib.__version__, @@ -79,8 +99,9 @@ setup(name='cmdtest', author='Lars Wirzenius', author_email='liw@liw.fi', url='http://liw.fi/cmdtest/', - scripts=['cmdtest'], + scripts=['cmdtest', 'yarn'], py_modules=['cmdtestlib'], + packages=['yarnlib'], data_files=[('share/man/man1', glob.glob('*.1'))], cmdclass={ 'build': GenerateManpage, diff --git a/shell-lib.sh b/shell-lib.sh new file mode 100644 index 0000000..767918e --- /dev/null +++ b/shell-lib.sh @@ -0,0 +1,6 @@ +# A shell library for the shell-lib.yarn test. + +implement() +{ + echo "$@" +} diff --git a/shell-lib.yarn b/shell-lib.yarn new file mode 100644 index 0000000..f9b9ca9 --- /dev/null +++ b/shell-lib.yarn @@ -0,0 +1,25 @@ +A simple test scenario with shell libraries +====================== + +This is a very simple test scenario, which exists only to test +the scenario test runner itself. + + SCENARIO a shell library scenario + +The following is the actual test in this scenario: + + GIVEN a given + WHEN a when + THEN a then + +And the implementations follow. + + IMPLEMENTS GIVEN a given + implement a given + + IMPLEMENTS WHEN a when + implement a when + + IMPLEMENTS THEN a then + implement a then + diff --git a/simple.scenario b/simple.scenario new file mode 100644 index 0000000..8061e47 --- /dev/null +++ b/simple.scenario @@ -0,0 +1,32 @@ +A simple test scenario +====================== + +This is a very simple test scenario, which exists only to test +the scenario test runner itself. + + SCENARIO a simple scenario + +The following is the actual test in this scenario: + + GIVEN a clean slate + WHEN nothing happens + THEN everything is OK + AND not all is well + FINALLY cleanup + +And the implementations follow. + + IMPLEMENTS GIVEN a clean slate + echo a clean slate! + + IMPLEMENTS WHEN nothing happens + true + + IMPLEMENTS THEN everything is OK + echo OK! + + IMPLEMENTS THEN not all is well + false + + IMPLEMENTS FINALLY cleanup + echo cleaning up diff --git a/without-tests b/without-tests new file mode 100644 index 0000000..6aa40f1 --- /dev/null +++ b/without-tests @@ -0,0 +1,3 @@ +yarnlib/__init__.py +setup.py +yarnlib/elements.py @@ -0,0 +1,343 @@ +#!/usr/bin/python +# Copyright 2013 Lars Wirzenius +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# =*= License: GPL-3+ =*= + + +import cliapp +import logging +import os +import re +import shutil +import sys +import tempfile +import time +import ttystatus + +import cmdtestlib +import yarnlib + + +class YarnRunner(cliapp.Application): + + def add_settings(self): + self.settings.boolean( + ['no-act', 'dry-run', 'pretend', 'n'], + 'do not actually run any tests, merely print what would be run') + + self.settings.boolean( + ['quiet', 'q'], + 'be quiet, avoid progress reporting, only show errors') + + self.settings.boolean( + ['verbose', 'v'], + 'make progress reporting be more verbose ("wall of text"), ' + 'instead of a one-line status info; this is turned ' + 'automatically if there is not terminal') + + self.settings.string_list( + ['shell-library', 's'], + 'include a shell library for the IMPLEMENTS sections to use') + + self.settings.string_list( + ['run', 'r'], + 'run only TEST (this option can be repeated)', + metavar='TEST') + + self.settings.string( + ['tempdir'], + 'use DIR as the temporary directory for tests; ' + 'it should be empty or not exist', + metavar='DIR') + + self.settings.boolean( + ['snapshot'], + 'make snapshots of test working directory ' + 'after each scenario step; you probably ' + 'want to use this with --tempdir') + + def info(self, msg): + if self.settings['verbose']: + logging.info(msg) + self.output.write('%s\n' % msg) + + def warning(self, msg): + if self.settings['verbose']: + logging.warning(msg) + self.output.write('WARNING: %s\n' % msg) + elif not self.settings['quiet']: + self.ts.notify('WARNING: %s' % msg) + + def error(self, msg): + if self.settings['verbose']: + logging.info(msg) + sys.stderr.write('%s\n' % msg) + elif not self.settings['quiet']: + self.ts.error(msg) + + def process_args(self, args): + # Do we have tty? If not, turn on --verbose, unless --quiet. + if not self.settings['quiet']: + try: + open('/dev/tty', 'w') + except IOError: + self.settings['verbose'] = True + + self.ts = ttystatus.TerminalStatus(period=0.001) + if not self.settings['quiet'] and not self.settings['verbose']: + self.ts.format( + '%ElapsedTime() %Index(scenario,scenarios): ' + '%String(scenario_name): ' + 'step %Index(step,steps): %String(step_name)') + + scenarios, implementations = self.parse_scenarios(args) + self.connect_implementations(scenarios, implementations) + shell_prelude = self.load_shell_libraries() + + self.ts['scenarios'] = scenarios + self.ts['num_scenarios'] = len(scenarios) + self.info('Found %d scenarios' % len(scenarios)) + + start_time = time.time() + failed_scenarios = [] + for scenario in self.select_scenarios(scenarios): + if not self.run_scenario(scenario, shell_prelude): + failed_scenarios.append(scenario) + duration = time.time() - start_time + + if not self.settings['quiet']: + self.ts.clear() + self.ts.finish() + + if failed_scenarios: + raise cliapp.AppException( + 'Test suite FAILED in %s scenarios' % len(failed_scenarios)) + + if not self.settings['quiet']: + print ( + 'Scenario test suite PASS, with %d scenarios, ' + 'in %.1f seconds' % + (len(scenarios), duration)) + + def parse_scenarios(self, filenames): + mdparser = yarnlib.MarkdownParser() + for filename in filenames: + self.info('Parsing scenario file %s' % filename) + blocks = mdparser.parse_file(filename) + if not blocks: + self.warning('No scenario code blocks in %s' % filename) + + block_parser = yarnlib.BlockParser() + block_parser.parse_blocks(mdparser.blocks) + + return block_parser.scenarios, block_parser.implementations + + def connect_implementations(self, scenarios, implementations): + for scenario in scenarios: + for step in scenario.steps: + self.connect_implementation(scenario, step, implementations) + + def connect_implementation(self, scenario, step, implementations): + matching = [i for i in implementations + if step.what == i.what and + re.match('(%s)$' % i.regexp, step.text, re.I)] + + if len(matching) == 0: + raise cliapp.AppException( + 'Scenario %s, step "%s %s" has no matching ' + 'implementation' % + (scenario.name, step.what, step.text)) + if len(matching) > 1: + s = '\n'.join( + 'IMPLEMENTS %s %s' % (i.what, i.regexp) + for i in matching) + raise cliapp.AppException( + 'Scenario "%s", step "%s %s" has more than one ' + 'matching implementations:\n%s' % + (scenario.name, step.what, step.text, s)) + + assert step.implementation is None + step.implementation = matching[0] + + def load_shell_libraries(self): + if not self.settings['shell-library']: + self.info('No shell libraries defined') + return '' + + libs = [] + for filename in self.settings['shell-library']: + self.info('Loading shell library %s' % filename) + with open(filename) as f: + text = f.read() + libs.append('# Loaded from %s\n\n%s\n\n' % (filename, text)) + + return ''.join(libs) + + def select_scenarios(self, scenarios): + + def normalise(s): + return ' '.join(s.lower().split()) + + def matches(a, b): + return normalise(a) == normalise(b) + + if self.settings['run']: + result = [] + for name in self.settings['run']: + for s in scenarios: + if matches(s.name, name) and s not in result: + result.append(s) + break + return result + + return scenarios + + def run_scenario(self, scenario, shell_prelude): + self.info('Running scenario %s' % scenario.name) + self.ts['scenario'] = scenario + self.ts['scenario_name'] = scenario.name + self.ts['steps'] = scenario.steps + + if self.settings['no-act']: + self.info('Pretending everything went OK') + return True + + if self.settings['tempdir']: + tempdir = self.settings['tempdir'] + if not os.path.exists(tempdir): + os.mkdir(tempdir) + else: + tempdir = tempfile.mkdtemp() + + os.mkdir(self.scenario_dir(tempdir, scenario)) + datadir = self.datadir(tempdir, scenario) + os.mkdir(datadir) + self.info('DATADIR is %s' % datadir) + + assuming = [s for s in scenario.steps if s.what == 'ASSUMING'] + cleanup = [s for s in scenario.steps if s.what == 'FINALLY'] + normal = [s for s in scenario.steps if s not in assuming + cleanup] + + ok = True + step_number = 0 + + for step in assuming: + exit = self.run_step(datadir, scenario, step, shell_prelude, False) + step_number += 1 + self.snapshot_datadir( + tempdir, datadir, scenario, step_number, step) + if exit != 0: + break + else: + for step in normal: + exit = self.run_step( + datadir, scenario, step, shell_prelude, True) + step_number += 1 + self.snapshot_datadir( + tempdir, datadir, scenario, step_number, step) + if exit != 0: + ok = False + break + + for step in cleanup: + exit = self.run_step( + datadir, scenario, step, shell_prelude, True) + step_number += 1 + self.snapshot_datadir( + tempdir, datadir, scenario, step_number, step) + if exit != 0: + ok = False + break + + if not self.settings['snapshot']: + shutil.rmtree(tempdir) + + return ok + + def run_step(self, datadir, scenario, step, shell_prelude, report_error): + self.info('Running step "%s %s"' % (step.what, step.text)) + self.ts['step'] = step + self.ts['step_name'] = '%s %s' % (step.what, step.text) + + m = re.match(step.implementation.regexp, step.text) + assert m is not None + env = os.environ.copy() + env['DATADIR'] = datadir + for i, match in enumerate(m.groups('')): + env['MATCH_%d' % (i+1)] = match + + shell_script = '%s\n\n%s\n' % ( + shell_prelude, step.implementation.shell) + exit, stdout, stderr = cliapp.runcmd_unchecked( + ['sh', '-euc', shell_script], env=env) + + logging.debug('Exit code: %d' % exit) + if stdout: + logging.debug('Standard output:\n%s' % self.indent(stdout)) + else: + logging.debug('Standard output: empty') + if stderr: + logging.debug('Standard error:\n%s' % self.indent(stderr)) + else: + logging.debug('Standard error: empty') + + if exit != 0 and report_error: + self.error( + 'ERROR: In scenario "%s"\nstep "%s %s" failed,\n' + 'with exit code %d:\n' + 'Standard output from shell command:\n%s' + 'Standard error from shell command:\n%s' % + (scenario.name, step.what, step.text, exit, + self.indent(stdout), self.indent(stderr))) + + return exit + + def scenario_dir(self, tempdir, scenario): + return os.path.join(tempdir, self.nice(scenario.name)) + + def datadir(self, tempdir, scenario): + sd = self.scenario_dir(tempdir, scenario) + return os.path.join(sd, 'datadir') + + def snapshot_dir(self, tempdir, scenario, step, step_number): + sd = self.scenario_dir(tempdir, scenario) + base = '%03d-%s-%s' % (step_number, step.what, self.nice(step.text)) + return os.path.join(sd, base) + + def snapshot_datadir(self, tempdir, datadir, scenario, step_number, step): + snapshot = self.snapshot_dir(tempdir, scenario, step, step_number) + cliapp.runcmd(['cp', '-a', datadir, snapshot]) + + def nice(self, name): + # Quote a scenario or step name so it forms a nice filename. + nice_chars = "abcdefghijklmnopqrstuvwxyz" + nice_chars += nice_chars.upper() + nice_chars += "0123456789-." + + nice = [] + for c in name: + if c in nice_chars: + nice.append(c) + elif not nice or nice[-1] != '_': + nice.append('_') + nice = ''.join(nice) + return nice + + def indent(self, s): + return ''.join(' %s\n' % line for line in s.splitlines()) + + +YarnRunner(version=cmdtestlib.__version__).run() diff --git a/yarn.1.in b/yarn.1.in new file mode 100644 index 0000000..85d2d8e --- /dev/null +++ b/yarn.1.in @@ -0,0 +1,137 @@ +.\" Copyright 2013 Lars Wirzenius <liw@liw.fi> +.\" +.\" This program is free software: you can redistribute it and/or modify +.\" it under the terms of the GNU General Public License as published by +.\" the Free Software Foundation, either version 3 of the License, or +.\" (at your option) any later version. +.\" +.\" This program is distributed in the hope that it will be useful, +.\" but WITHOUT ANY WARRANTY; without even the implied warranty of +.\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +.\" GNU General Public License for more details. +.\" +.\" You should have received a copy of the GNU General Public License +.\" along with this program. If not, see <http://www.gnu.org/licenses/>. +.\" +.TH YARN 1 +.SH NAME +yarn \- scenario testing of Unix command line tools +.SH SYNOPSIS +.SH DESCRIPTION +.B yarn +is a scenario testing tool: +you write a scenario describing how a user uses your software +and what should happen, +and express, +using very lightweight syntax, +the scenario in such a way that it can be tested automatically. +The scenario has a simple, but strict structure: +.IP +.nf +GIVEN some setup for the test +WHEN thing that is to be tested happens +THEN the post-conditions must be true +.fi +.PP +As an example, consider a very short test scenario for verifying that +a backup program works, at least for one simple case. +.IP +.nf +SCENARIO backups can be restored +GIVEN some live data in a directory +AND an empty backup repository +WHEN a backup is made +THEN the data case be restored +FINALLY cleanup +.fi +.PP +Note the addition of AND: you can have multiple GIVEN, WHEN, and +THEN statements. The AND keyword makes the text be more readable. +SCENARIO is also necessary, and gives the title. +.PP +FINALLY is for cleanups. +The FINALLY steps will be run regardless of whether the scenario succeeds +or not. +.PP +Scenarios are meant to be written in somewhat human readable language. +However, they are not free form text. +In addition to the GIVEN/WHEN/THEN structure, +the text for each of the steps needs a computer-executable implementation. +This is done by using IMPLEMENTS. +The backup scenario from above might be implemented as follows: +.IP +.nf +IMPLEMENTS GIVEN some live data in a directory +rm -rf "$TESTDIR/data" +mkdir "$TESTDIR/data" +echo foo > "$TESTDIR/data/foo" +.IP +IMPLEMENTS GIVEN an empty backup repository +rm -rf "$TESTDIR/repo" +mkdir "$TESTDIR/repo" +.IP +IMPLEMENTS WHEN a backup is made +backup-program -r "$TESTDIR/repo" "$TESTDIR/data" +.IP +IMPLEMENTS THEN the data can be restored +mkdir "$TESTDIR/restored" +restore-program -r "$TESTDIR/repo" "$TESTDIR/restored" +diff -rq "$TESTDIR/data" "$TESTDIR/restored" +.IP +IMPLEMENTS FINALLY cleanup +echo nothing to do, actually +.fi +.PP +Each "IMPLEMENTS GIVEN" (or WHEN, THEN, FINALLY) is followed by a regular +expression on the same line, +and then a shell script that gets executed to implement any step +that matches the regular expression. +The implementation can extract data from the match as well: +for example, the regular expression might allow a file size to be specified. +.PP +The above example is a bit silly, of course: +why go to the effort to obfuscate the various steps? +The answer is that the various steps, +implemented using IMPLEMENTS, +can be combined in many ways, +to test different aspects of the program being tested. +.PP +Moreover, +by making the step descriptions be human language text, +matched by regular expressions, +most of the test can hopefully be written, +and understood, +by non-programmers. +Someone who understands what a program should do, +could write tests to verify its behaviour. +The implementations of the various steps need to be implemented +by a programmer, +but given a well-designed set of steps, +with enough flexibility in their implementation, +that quite a good test suite can be written. +.SH OPTIONS +.SH EXAMPLE +To run +.B yarn +on all the scenarios in your current directory: +.IP +.nf +yarn *.scenario +.fi +.PP +All the files will be treated together as if they had been one file. +.PP +To add a shell library to be included when running any IMPLEMENTS section: +.IP +.nf +yarn \-\-shell\-library mylib.sh *.scenario +.fi +.PP +You can repeat +.B \-\-shell\-library +as many times as necessary. +.SH "SEE ALSO" +.BR cmdtest (1), +.BR cliapp (5). +.PP +The README.yarn file has more details on the scenario testing language. diff --git a/yarn.tests/assuming-failure.script b/yarn.tests/assuming-failure.script new file mode 100755 index 0000000..9752e39 --- /dev/null +++ b/yarn.tests/assuming-failure.script @@ -0,0 +1,23 @@ +#!/bin/sh + +set -eu + + +cat <<EOF > "$DATADIR/test.yarn" + SCENARIO foo + ASSUMING something + THEN remember + FINALLY cleanup + + IMPLEMENTS ASSUMING something + false + IMPLEMENTS THEN remember + touch "$DATADIR/then-flag" + IMPLEMENTS FINALLY cleanup + touch "$DATADIR/cleanup-flag" +EOF + +./run-yarn "$DATADIR/test.yarn" +[ ! -e "$DATADIR/then-flag" ] +[ ! -e "$DATADIR/cleanup-flag" ] + diff --git a/yarn.tests/assuming.script b/yarn.tests/assuming.script new file mode 100755 index 0000000..52eee36 --- /dev/null +++ b/yarn.tests/assuming.script @@ -0,0 +1,23 @@ +#!/bin/sh + +set -eu + + +cat <<EOF > "$DATADIR/test.yarn" + SCENARIO foo + ASSUMING something + THEN remember + FINALLY cleanup + + IMPLEMENTS ASSUMING something + true + IMPLEMENTS THEN remember + touch "$DATADIR/then-flag" + IMPLEMENTS FINALLY cleanup + touch "$DATADIR/cleanup-flag" +EOF + +./run-yarn "$DATADIR/test.yarn" +[ -e "$DATADIR/then-flag" ] +[ -e "$DATADIR/cleanup-flag" ] + diff --git a/yarn.tests/finally.script b/yarn.tests/finally.script new file mode 100755 index 0000000..4e05d2e --- /dev/null +++ b/yarn.tests/finally.script @@ -0,0 +1,26 @@ +#!/bin/sh + +set -eu + +cat <<EOF > "$DATADIR/finally.yarn" + SCENARIO finally + GIVEN nothing + WHEN nothing + THEN nothing + FINALLY yeehaa + + IMPLEMENTS GIVEN nothing + true + + IMPLEMENTS WHEN nothing + true + + IMPLEMENTS THEN nothing + true + + IMPLEMENTS FINALLY yeehaa + touch "$DATADIR/finally.has.run" +EOF + +./run-yarn "$DATADIR/finally.yarn" +test -e "$DATADIR/finally.has.run" diff --git a/yarn.tests/multi.script b/yarn.tests/multi.script new file mode 100755 index 0000000..c8131da --- /dev/null +++ b/yarn.tests/multi.script @@ -0,0 +1,23 @@ +#!/bin/sh + +set -eu + + +cat <<EOF > "$DATADIR/1.yarn" + SCENARIO foo + GIVEN all is ok + WHEN doing ok + THEN be ok +EOF + +cat <<EOF > "$DATADIR/2.yarn" + IMPLEMENTS GIVEN all is ok + true + IMPLEMENTS WHEN doing ok + true + IMPLEMENTS THEN be ok + true +EOF + +./run-yarn "$DATADIR/1.yarn" "$DATADIR/2.yarn" | +sed 's/, in .* seconds$//' diff --git a/yarn.tests/no-act.script b/yarn.tests/no-act.script new file mode 100755 index 0000000..cc924da --- /dev/null +++ b/yarn.tests/no-act.script @@ -0,0 +1,22 @@ +#!/bin/sh + +set -eu + +# Create a scenario that will fail. +cat <<EOF > "$DATADIR/fail.yarn" + SCENARIO this will fail + GIVEN badness + WHEN bad things happen + THEN more badness + + IMPLEMENTS GIVEN badness + false + + IMPLEMENTS WHEN bad things happen + false + + IMPLEMENTS THEN more badness + false +EOF + +./run-yarn -n "$DATADIR/fail.yarn" diff --git a/yarn.tests/selected-test.script b/yarn.tests/selected-test.script new file mode 100755 index 0000000..2c4521d --- /dev/null +++ b/yarn.tests/selected-test.script @@ -0,0 +1,18 @@ +#!/bin/sh + +set -eu + +cat << EOF > "$DATADIR/test.yarn" + SCENARIO bar test + THEN do bar + + SCENARIO foo test + THEN do foo + + IMPLEMENTS THEN do (.*) + touch "$DATADIR/\$MATCH_1" +EOF + +./run-yarn "$DATADIR/test.yarn" --run 'foo test' +test -e "$DATADIR/foo" +! test -e "$DATADIR/bar" diff --git a/yarn.tests/setup b/yarn.tests/setup new file mode 100755 index 0000000..d64c24e --- /dev/null +++ b/yarn.tests/setup @@ -0,0 +1,3 @@ +#!/bin/sh + +find "$DATADIR" -mindepth 1 -delete diff --git a/yarn.tests/shell-lib.script b/yarn.tests/shell-lib.script new file mode 100755 index 0000000..68e4eb1 --- /dev/null +++ b/yarn.tests/shell-lib.script @@ -0,0 +1,5 @@ +#!/bin/sh + +set -eu + +./run-yarn --shell-library shell-lib.sh shell-lib.yarn diff --git a/yarn.tests/simple.exit b/yarn.tests/simple.exit new file mode 100644 index 0000000..d00491f --- /dev/null +++ b/yarn.tests/simple.exit @@ -0,0 +1 @@ +1 diff --git a/yarn.tests/simple.script b/yarn.tests/simple.script new file mode 100755 index 0000000..70d6103 --- /dev/null +++ b/yarn.tests/simple.script @@ -0,0 +1,5 @@ +#!/bin/sh + +set -eu + +./run-yarn simple.scenario diff --git a/yarn.tests/simple.stderr b/yarn.tests/simple.stderr new file mode 100644 index 0000000..df917a0 --- /dev/null +++ b/yarn.tests/simple.stderr @@ -0,0 +1 @@ +ERROR: Test suite FAILED in 1 scenarios diff --git a/yarn.tests/snapshot.script b/yarn.tests/snapshot.script new file mode 100755 index 0000000..def73a8 --- /dev/null +++ b/yarn.tests/snapshot.script @@ -0,0 +1,62 @@ +#!/bin/sh + +set -eu + +cat << EOF > "$DATADIR/foo.yarn" + SCENARIO foo + GIVEN foo + WHEN foo + THEN foo + + SCENARIO bar + GIVEN bar + WHEN bar + THEN bar + + IMPLEMENTS GIVEN (.*) + touch "\$DATADIR/\$MATCH_1.given" + + IMPLEMENTS WHEN (.*) + touch "\$DATADIR/\$MATCH_1.when" + + IMPLEMENTS THEN (.*) + touch "\$DATADIR/\$MATCH_1.then" +EOF + +./run-yarn --snapshot --tempdir "$DATADIR/tmp" "$DATADIR/foo.yarn" + +test -e "$DATADIR/tmp/bar" +test -e "$DATADIR/tmp/bar/datadir" +test -e "$DATADIR/tmp/bar/datadir/bar.given" +test -e "$DATADIR/tmp/bar/datadir/bar.when" +test -e "$DATADIR/tmp/bar/datadir/bar.then" + +test -e "$DATADIR/tmp/bar/001-GIVEN-bar" +test -e "$DATADIR/tmp/bar/001-GIVEN-bar/bar.given" +! test -e "$DATADIR/tmp/bar/001-GIVEN-bar/bar.when" +! test -e "$DATADIR/tmp/bar/001-GIVEN-bar/bar.then" + +test -e "$DATADIR/tmp/bar/002-WHEN-bar" +test -e "$DATADIR/tmp/bar/002-WHEN-bar/bar.given" +test -e "$DATADIR/tmp/bar/002-WHEN-bar/bar.when" +! test -e "$DATADIR/tmp/bar/002-WHEN-bar/bar.then" + +test -e "$DATADIR/tmp/bar/003-THEN-bar" +test -e "$DATADIR/tmp/bar/003-THEN-bar/bar.given" +test -e "$DATADIR/tmp/bar/003-THEN-bar/bar.when" +test -e "$DATADIR/tmp/bar/003-THEN-bar/bar.then" + +test -e "$DATADIR/tmp/foo/001-GIVEN-foo" +test -e "$DATADIR/tmp/foo/001-GIVEN-foo/foo.given" +! test -e "$DATADIR/tmp/foo/001-GIVEN-foo/foo.when" +! test -e "$DATADIR/tmp/foo/001-GIVEN-foo/foo.then" + +test -e "$DATADIR/tmp/foo/002-WHEN-foo" +test -e "$DATADIR/tmp/foo/002-WHEN-foo/foo.given" +test -e "$DATADIR/tmp/foo/002-WHEN-foo/foo.when" +! test -e "$DATADIR/tmp/foo/002-WHEN-foo/foo.then" + +test -e "$DATADIR/tmp/foo/003-THEN-foo" +test -e "$DATADIR/tmp/foo/003-THEN-foo/foo.given" +test -e "$DATADIR/tmp/foo/003-THEN-foo/foo.when" +test -e "$DATADIR/tmp/foo/003-THEN-foo/foo.then" diff --git a/yarn.tests/warn-if-empty.script b/yarn.tests/warn-if-empty.script new file mode 100755 index 0000000..03be52d --- /dev/null +++ b/yarn.tests/warn-if-empty.script @@ -0,0 +1,10 @@ +#!/bin/sh + +set -eu + +touch "$DATADIR/empty.yarn" + +# The grep below will fail unless the string exists, thereby failing the +# entire test. +./run-yarn --no-quiet --log=/dev/stdout "$DATADIR/empty.yarn" 2>&1 | + grep 'No scenario code blocks' > /dev/null diff --git a/yarnlib/__init__.py b/yarnlib/__init__.py new file mode 100644 index 0000000..88bf46f --- /dev/null +++ b/yarnlib/__init__.py @@ -0,0 +1,21 @@ +# Copyright 2013 Lars Wirzenius +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# =*= License: GPL-3+ =*= + + +from mdparser import MarkdownParser +from elements import Scenario, ScenarioStep, Implementation +from block_parser import BlockParser, BlockError diff --git a/yarnlib/block_parser.py b/yarnlib/block_parser.py new file mode 100644 index 0000000..64dc990 --- /dev/null +++ b/yarnlib/block_parser.py @@ -0,0 +1,142 @@ +# Copyright 2013 Lars Wirzenius +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# =*= License: GPL-3+ =*= + + +import cliapp + +import yarnlib + + +class BlockError(cliapp.AppException): + + pass + + +# Parse a sequence of textual blocks into scenario and Implementation +# objects, and their constituent objects. + +class BlockParser(object): + + def __init__(self): + self.scenarios = [] + self.implementations = [] + self.line_parsers = { + 'SCENARIO': self.parse_scenario, + 'ASSUMING': self.parse_assuming, + 'GIVEN': self.parse_given, + 'WHEN': self.parse_when, + 'THEN': self.parse_then, + 'FINALLY': self.parse_finally, + 'AND': self.parse_and, + 'IMPLEMENTS': self.parse_implementing, + } + + def parse_blocks(self, blocks): + while blocks: + blocks = self.parse_one(blocks) + + def parse_one(self, blocks): + assert blocks + block = blocks[0] + assert block + t = block.split('\n', 1) + assert len(t) in [1,2] + if len(t) == 1: + line1 = block + block = '' + else: + line1, block = t + if block: + blocks[0] = block + else: + del blocks[0] + + words = line1.split() + if not words: + return blocks + rest = ' '.join(words[1:]) + + for keyword in self.line_parsers: + if words[0] == keyword: + return self.line_parsers[keyword](rest, blocks) + + raise BlockError("Syntax error: unknown step: %s" % line1) + + def parse_scenario(self, line, blocks): + self.scenarios.append(yarnlib.Scenario(line)) + return blocks + + def parse_simple(self, what, line, blocks): + if not self.scenarios: + raise BlockError('Syntax errror: %s before SCENARIO' % what) + step = yarnlib.ScenarioStep(what, line) + self.scenarios[-1].steps.append(step) + return blocks + + def parse_assuming(self, line, blocks): + return self.parse_simple('ASSUMING', line, blocks) + + def parse_given(self, line, blocks): + return self.parse_simple('GIVEN', line, blocks) + + def parse_when(self, line, blocks): + return self.parse_simple('WHEN', line, blocks) + + def parse_then(self, line, blocks): + return self.parse_simple('THEN', line, blocks) + + def parse_finally(self, line, blocks): + return self.parse_simple('FINALLY', line, blocks) + + def parse_and(self, line, blocks): + if not self.scenarios: + raise BlockError('Syntax errror: AND before SCENARIO') + scenario = self.scenarios[-1] + if not scenario.steps: + raise BlockError( + 'Syntax errror: AND before what it would continue') + step = scenario.steps[-1] + assert step.what in self.line_parsers + return self.line_parsers[step.what](line, blocks) + + def parse_implementing(self, line, blocks): + words = line.split() + if len(words) < 2: + raise BlockError( + 'Syntax error: IMPLEMENTS must have what and regexp') + what = words[0] + regexp = ' '.join(words[1:]) + if blocks: + block = blocks[0] + shell = [] + rest = [] + for block_line in block.splitlines(): + if rest or block_line.startswith('IMPLEMENTS'): + rest.append(block_line) + else: + shell.append(block_line) + shell = '\n'.join(shell) + if rest: + blocks[0] = '\n'.join(rest) + else: + del blocks[0] + else: + shell = '' + implementation = yarnlib.Implementation(what, regexp, shell) + self.implementations.append(implementation) + return blocks + diff --git a/yarnlib/block_parser_tests.py b/yarnlib/block_parser_tests.py new file mode 100644 index 0000000..78754b5 --- /dev/null +++ b/yarnlib/block_parser_tests.py @@ -0,0 +1,134 @@ +# Copyright 2013 Lars Wirzenius +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# =*= License: GPL-3+ =*= + + +import unittest + +import yarnlib + + +class BlockParserTests(unittest.TestCase): + + def setUp(self): + self.parser = yarnlib.BlockParser() + + def test_is_initially_empty(self): + self.assertEqual(self.parser.scenarios, []) + self.assertEqual(self.parser.implementations, []) + + def test_parses_simple_elements(self): + self.parser.parse_blocks( + ['SCENARIO foo', 'ASSUMING something', 'GIVEN bar', + 'WHEN foobar\nTHEN yoyo\nFINALLY yay\nAND yeehaa']) + + self.assertEqual(len(self.parser.scenarios), 1) + self.assertEqual(len(self.parser.implementations), 0) + + scenario = self.parser.scenarios[0] + self.assertEqual(scenario.name, 'foo') + self.assertEqual(len(scenario.steps), 6) + self.assertEqual(scenario.steps[0].what, 'ASSUMING') + self.assertEqual(scenario.steps[0].text, 'something') + self.assertEqual(scenario.steps[1].what, 'GIVEN') + self.assertEqual(scenario.steps[1].text, 'bar') + self.assertEqual(scenario.steps[2].what, 'WHEN') + self.assertEqual(scenario.steps[2].text, 'foobar') + self.assertEqual(scenario.steps[3].what, 'THEN') + self.assertEqual(scenario.steps[3].text, 'yoyo') + self.assertEqual(scenario.steps[4].what, 'FINALLY') + self.assertEqual(scenario.steps[4].text, 'yay') + self.assertEqual(scenario.steps[5].what, 'FINALLY') + self.assertEqual(scenario.steps[5].text, 'yeehaa') + + def test_normalises_whitespace(self): + self.parser.parse_blocks(['SCENARIO foo bar ']) + self.assertEqual(self.parser.scenarios[0].name, 'foo bar') + + def test_handles_empty_line(self): + self.parser.parse_blocks(['SCENARIO foo\n\nGIVEN bar\nTHEN foobar']) + self.assertEqual(len(self.parser.scenarios), 1) + + def test_raises_error_for_unknown_step(self): + self.assertRaises( + yarnlib.BlockError, + self.parser.parse_blocks, + ['SCENARIO foo\nblah']) + + def test_raises_error_for_step_outside_scenario(self): + self.assertRaises( + yarnlib.BlockError, + self.parser.parse_blocks, + ['GIVEN foo']) + + def test_raises_error_for_AND_before_scenario(self): + self.assertRaises( + yarnlib.BlockError, + self.parser.parse_blocks, + ['AND bar']) + + def test_raises_error_for_AND_before_step(self): + self.assertRaises( + yarnlib.BlockError, + self.parser.parse_blocks, + ['SCENARIO foo\nAND bar']) + + def test_parses_implements_in_a_block_by_itself(self): + self.parser.parse_blocks(['IMPLEMENTS GIVEN foo\ntrue']) + impls = self.parser.implementations + self.assertEqual(len(impls), 1) + self.assertEqual(impls[0].what, 'GIVEN') + self.assertEqual(impls[0].regexp, 'foo') + self.assertEqual(impls[0].shell, 'true') + + def test_parses_implements_with_empty_shell_text(self): + self.parser.parse_blocks(['IMPLEMENTS GIVEN foo']) + impls = self.parser.implementations + self.assertEqual(len(impls), 1) + self.assertEqual(impls[0].what, 'GIVEN') + self.assertEqual(impls[0].regexp, 'foo') + self.assertEqual(impls[0].shell, '') + + def test_parses_two_implements_in_a_code_block(self): + self.parser.parse_blocks( + ['IMPLEMENTS GIVEN foo\ntrue\nIMPLEMENTS WHEN bar\ncat /dev/null']) + impls = self.parser.implementations + self.assertEqual(len(impls), 2) + self.assertEqual(impls[0].what, 'GIVEN') + self.assertEqual(impls[0].regexp, 'foo') + self.assertEqual(impls[0].shell, 'true') + self.assertEqual(impls[1].what, 'WHEN') + self.assertEqual(impls[1].regexp, 'bar') + self.assertEqual(impls[1].shell, 'cat /dev/null') + + def test_raises_error_for_implements_with_no_args(self): + self.assertRaises( + yarnlib.BlockError, + self.parser.parse_blocks, + ['IMPLEMENTS']) + + def test_raises_error_for_implements_with_one_args(self): + self.assertRaises( + yarnlib.BlockError, + self.parser.parse_blocks, + ['IMPLEMENTS GIVEN']) + + def test_raises_error_for_implements_with_first_args_not_a_keyword(self): + self.assertRaises( + yarnlib.BlockError, + self.parser.parse_blocks, + ['IMPLEMENTS foo']) + diff --git a/yarnlib/elements.py b/yarnlib/elements.py new file mode 100644 index 0000000..9eeb1b7 --- /dev/null +++ b/yarnlib/elements.py @@ -0,0 +1,51 @@ +# Copyright 2013 Lars Wirzenius +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# =*= License: GPL-3+ =*= + + +# This is a step in a scenario: GIVEN, WHEN, THEN, etc. + +class ScenarioStep(object): + + def __init__(self, what, text): + self.what = what + self.text = text + self.implementation = None + + +# This is the scenario itself. + +class Scenario(object): + + def __init__(self, name): + self.name = name + self.steps = [] + + +# This is an IMPLEMENTS chunk. + +class Implementation(object): + + def __init__(self, what, regexp, shell): + self.what = what + self.regexp = regexp + self.shell = shell + + def execute(self): + exit, out, err = cliapp.runcmd_unchecked( + ['sh', '-c', 'set -eu\n' + self.shell]) + return exit + diff --git a/yarnlib/mdparser.py b/yarnlib/mdparser.py new file mode 100644 index 0000000..f787293 --- /dev/null +++ b/yarnlib/mdparser.py @@ -0,0 +1,76 @@ +# Copyright 2013 Lars Wirzenius +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# =*= License: GPL-3+ =*= + + +import logging +import markdown +import StringIO +from markdown.treeprocessors import Treeprocessor + + +# +# Classes for Markdown parsing. See python-markdown documentation +# for details. We want to find all top level code blocks (indented +# four spaces in the Markdown), which we'll parse for scenario test +# stuff later on. We create a Python markdown extension and use +# "tree processor" to analyse the parsed ElementTree at the right +# moment for top level <pre> blocks. +# + +# This is a Treeprocessor that iterates over the parsed Markdown, +# as an ElementTree, and finds all top level code blocks. + +class GatherCodeBlocks(Treeprocessor): + + def __init__(self, blocks): + self.blocks = blocks + + def run(self, root): + for child in root.getchildren(): + if child.tag == 'pre': + code = child.find('code') + self.blocks.append(code.text) + return root + +# This is the Python Markdown extension to call the code block +# gatherer at the right time. It stores the list of top level +# code blocks as the blocks attribute. + +class ParseScenarioTestBlocks(markdown.extensions.Extension): + + def extendMarkdown(self, md, md_globals): + self.blocks = [] + self.gatherer = GatherCodeBlocks(self.blocks) + md.treeprocessors.add('gathercode', self.gatherer, '_end') + + +class MarkdownParser(object): + + def __init__(self): + self.blocks = [] + + def parse_string(self, text): + ext = ParseScenarioTestBlocks() + f = StringIO.StringIO() + markdown.markdown(text, output=f, extensions=[ext]) + self.blocks.extend(ext.blocks) + return ext.blocks + + def parse_file(self, filename): # pragma: no cover + with open(filename) as f: + return self.parse_string(f.read()) + diff --git a/yarnlib/mdparser_tests.py b/yarnlib/mdparser_tests.py new file mode 100644 index 0000000..099ff2b --- /dev/null +++ b/yarnlib/mdparser_tests.py @@ -0,0 +1,96 @@ +# Copyright 2013 Lars Wirzenius +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# =*= License: GPL-3+ =*= + + +import unittest + +import yarnlib + + +class MarkdownParserTests(unittest.TestCase): + + def setUp(self): + self.parser = yarnlib.MarkdownParser() + + def test_finds_code_block(self): + result = self.parser.parse_string(''' +This is blah blah text. + + this is a code block + +More text. +''') + self.assertEqual(self.parser.blocks, ['this is a code block\n']) + self.assertEqual(result, ['this is a code block\n']) + + def test_finds_consecutive_code_blocks_as_one(self): + self.parser.parse_string(''' +This is blah blah text. + + this is a code block + + this is a second code block + +More text. +''') + self.assertEqual( + self.parser.blocks, + ['this is a code block\n\nthis is a second code block\n']) + + def test_finds_code_blocks_with_text_in_between_as_two_blocks(self): + self.parser.parse_string(''' +This is blah blah text. + + this is a code block + +Blah. + + this is a second code block + +More text. +''') + self.assertEqual( + self.parser.blocks, + ['this is a code block\n', 'this is a second code block\n']) + + def test_only_finds_top_level_code_blocks(self): + self.parser.parse_string(''' +This is blah blah text. + + this is a code block + +And now a list: + +* list item + + this is a second level code block + +More text. +''') + self.assertEqual(self.parser.blocks, ['this is a code block\n']) + + def test_parses_multiple_files(self): + result1 = self.parser.parse_string(''' + block 1 +''') + result2 = self.parser.parse_string(''' + block 2 +''') + self.assertEqual(result1, ['block 1\n']) + self.assertEqual(result2, ['block 2\n']) + self.assertEqual(self.parser.blocks, ['block 1\n', 'block 2\n']) + |