summaryrefslogtreecommitdiff
path: root/ocamltest/ocamltest.org
blob: f547a0f2c812b120d22eab3fb28693387a0cb4ed (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
#+STARTUP: showall

#+title: The ocamltest reference manual
#+language: en

#+HTML_HEAD: <style> body { font-size: 1rem; max-width: 900px; margin: 0 auto; } </style>

* Introduction

This is =verbatim= and this is ~code~.

** What is ocamltest

ocamltest is a test-driver, that is, a program that can run tests and report
their results so that they can be used by a test infrastructure.

Originally, the tool has been designed specifically to run the integration
tests of the OCaml compiler's test suite. However, it has been
designed with extensibility in mind and thus has a plugin
mechanism that makes it possible to extend it with other tests.

** Design choices

*** Programming language and external dependencies

For a start, one may wonder in which language a test-driver for a compiler
should be written. It may indeed seem odd to write a test-driver for a
compiler in the language it compiles, since the compiler itself
is yet untested and thus not trustworthy.

It can however be observed that the OCaml compiler is /bootstrapped/,
meaning that it is itself written in OCaml. A newer version of the
compiler can thus be produced from an existing one and the (OCaml)
source code of that newer version. Practically, this means that the
compiler works at least well enough to recompile itself. This is why we
consider that it is okay to write a test-driver like ocamltest in OCaml,
as long as it uses only code that has been used to bootstrap the
compiler. In particular, this is why we prefer not to rely on any
external dependency, not even the libraries included in the compiler
distribution such as the =Unix= library.

*** Test types

As has been noted above, ocamltest has been developed to run the already
existing integration tests of the OCaml compiler's test suite, which
were previously run by a set of makefiles. This context explains
several design decisions which could otherwise seem rather arbitrary.

For example, the reason why ocamltest has no support for running unit tests
is that there were no such tests in the OCaml compiler's test suite.

Indeed, the OCaml compiler's test suite is composed mainly of complete
programs. In this context, the most usual meaning of "testing" a program
is that the program needs to be compiled and executed. The test will
be considered successful if the program compiles as expected and, when run,
returns the expected value.

Since this scenario is the most frequent one, it was of particular
importance to make writing tests of this form as simple as possible.

However, not all tests fall into the previously described category, so it is
also necessary to support not only variations on the previous scenario
(compile but do not run, compile with certain options, etc.) but also
completely different tests, such as REPL tests, debugger tests,
etc.

To fulfill these requirements and make it as easy as possible to turn a
program into a test, it has been chosen to design a Domain-Specific
Language (DSL) used to annotate the test program with a
=(* TEST *)= block at its top. This block specifies how the test
should be performed.

** Outline of this document

The next chapter explains through examples how to write simple tests. We
then introduce the key concepts used by ocamltest to provide a better
understanding of how it works and can be used to write more complex
tests. The last two chapters give an in-depth description of the
built-in tests and actions and of the tests and actions that are specific
to the OCaml compiler.

* Writing simple tests

This chapter is a tutorial. It explains how to write simple test
programs and also tries to give insights about how ocamltest works. These
insights will be deepened in chapter [[#concepts]] where ocamltest is
presented in a more abstract and conceptual way.

We start by explaining how to set-up a proper environment for writing
tests. We then show how to turn the traditional "Hello, world!" program
into a test and explain how to run it with ocamltest. We continue
with a few variations on this test and conclude this chapter
with a few other useful tests.

** Prerequisites for writing tests

Writing tests requires that the sources of the OCaml compiler for which
one wants to write them are downloaded and compiled. The compiler
does not need to be installed.

The sources can be downloaded either as an archive, or directly cloned
through git, which seems more appropriate in the context of writing ones
own tests. Refer to
=INSTALL.adoc= (and also to =README.win32.adoc= if you are on Windows) to
learn how to get the sources of the OCaml compiler and to compile them.

In the remainder of this manual, we will assume that the sources of the
OCaml compiler have been extracted in the =${OCAMLSRCDIR}= directory (for
instance =${HOME}/src/ocaml=) and that you have successfully configured
and compiled them as described in =INSTALL.adoc= or =README.win32.adoc=,
according to your operating system. The tools and libraries necessary
for running tests should also be built. This can be achieved by running
the following command from =${OCAMLSRCDIR}=:
: make -C testsuite lib tools

We will also assume that an =ocamltest= command is available in
your =PATH=. Although this is not strictly necessary, it is strongly
recommended that you set this up because this will simplify test
development a lot. This can be achieved e.g. by creating a symbolic
link to =${OCAMLSRCDIR}/ocamltest/ocamltest= (or its native
counterpart =${OCAMLSRCDIR}/ocamltest/ocamltest.opt=) in a directory that
is already in your =PATH=, like =~/bin=.

** Testing the "Hello, world!" program with the default tests

*** Turning "Hello, world!" into a useful test program

Consider the following OCaml implementation of the classical "Hello, world!"
program written to a =hello.ml= file:

: let _ = print_endline "Hello, world!"

Now assume we would like to make sure that the OCaml compiler can
compile this program and that the resulting executable indeed prints the
expected output. Here are the required steps to turn the program
above into a test usable by ocamltest to verify this:

1. First, we add a special comment at the very beginning of our =hello.ml=
   file to make it explicit that it is a test:
   #+begin_src
   (* TEST *)

   let _ = print_endline "Hello, world!"
   #+end_src

2. We then need to say what the expected outputs are. In our case, we
   expect that compiling the test produces no output at all and that its
   execution produces one single line:
   : Hello, world!
   To let ocamltest know about this, we create a =hello.reference= file
   containing the program's expected output -- the line mentioned
   above. There is nothing special to do for silent compilations
   since this is what is expected by default and a non-silent
   compilation would actually cause a test failure.

3. We can now ask ocamltest to run our test program with the
   following command:
   : ocamltest hello.ml

   Running this would produce an output similar to this one:

   #+begin_src
    ... testing 'hello.ml' with 1 (native) => passed
    ... testing 'hello.ml' with 2 (bytecode) => passed
   #+end_src

   In addition to this output, it may be noticed that the previous
   command has also created an =_ocamltest= directory whose content will
   be examined in the next sub-section.

4. Finally, there is one extra step required if we want our newly created
   test to be run automatically as part of the OCaml compiler's test suite.
   We need to move =hello.ml= and =hello.reference= to a directory (say
   =newtest=) located somewhere below =testsuite/tests= in the compiler's
   source tree. Once this is done, the command
   : make all
   executed in the =testsuite= directory of the OCaml compiler' source
   tree will run all the test suite, which now also includes our own test.

*** What exactly is going on during the test

The only thing we know from ocamltest's output when run on =hello.ml= is
that it is running two tests named =bytecode= and =native= and that the two of
them succeed. This can seem rather uninformative, and in a way it is, but
it has to be kept in mind that this information is the one passed by the
test-driver (ocamltest) to the test infrastructure. In that respect,
this is enough. For us users, though, it is not. That's why
ocamltest logs much more details about what is going on in a per-test
log file, which should be located in the =_ocamltest/hello/hello.log= file
found in the directory where =hello.ml= is.

Before looking at this log file, notice that it has been created in a
test-specific directory. ocamltest creates such a directory for each
file it tests and makes sure every file produced as a result of
testing this file will be placed in this directory, either directly, or
in one of its sub-directories. The latter happens if the test has
to be compiled several times, with the same compiler and different
command-line options, or with different compilers. In particular,
in order to better understand what follows, it may be helpful to
remember that =OCaml= actually consists in not less than four compilers:
=ocamlc.byte= and =ocamlc.opt= which are the bytecode and native
flavors of the bytecode compiler and =ocamlopt.byte= and
=ocamlopt.opt= which are the bytecode and native flavors of the native
compiler. So, as we will see, ''testing the bytecode compiler''
actually involves testing two compilers, and the same goes for ''testing
the native compiler''.

Now that all this has been spelled out, let's examine the log file
produced by the test. Although it is too long to be reproduced here,
it is recommended to go through it quickly to get an idea of its
structure. Here is how it starts:

#+begin_src
Specified modules: hello.ml
Source modules: hello.ml
#+end_src

The first line lists the names of the modules the test consists of. The
second line is almost similar but if some modules had separate
interface files, they would be listed here, too, without the user
having to specify them in the list of modules (for each specified =.ml=
file, ocamltest looks whether a corresponding =.mli= file exists and, if
so, adds it to the list of files to consider).

The rest of the log file can be split into two parts which are very
similar to each other: one for the =native= test and one for the =bytecode=
test. Among other things, we learn that each of these tests is composed
of nine actions. Before diving into the details of what each of these
actions does, let us take this opportunity to introduce a bit of
ocamltest terminology. An /action/ is anything that can =pass=, =skip= or
=fail=. A =test= is a sequence of such actions. Running
a test thus means running each of its actions, in sequence, until all the
actions have been run or one of them returns =fail= or =skip=. Whatever
the last run action returns, this value will be the result of the whole
test.

To give concrete examples of actions, let's briefly go over the nine ones
involved in the =bytecode= test (those for the =native= test are
quite similar):

1. =setup-ocamlc.byte-build-env=:: as its name suggests, this action
   creates a build environment where a program can be compiled and
   executed using the =ocamlc.byte= compiler. More precisely, this
   involves creating a dedicated directory under the test-file specific
   directory and populating it with the files required by subsequent actions.
   Depending on what the underlying operating system supports, the files
   will be either symlinked or copied from the test source directory.

2. =ocamlc.byte=:: invokes the =ocamlc.byte= compiler in various ways.
   Here, the test program is compiled and linked, but as we will see
   later, different behaviors are possible depending on ocamltest
   /variables/.

3. =check-ocamlc.byte-output=:: this action compares the compiler's
   output to a reference file, if one exists. As has been mentioned
   earlier, the absence of such a reference file specifies that the
   compiler's output is expected to be empty -- if it is not, this
   causes a failure of this action and thus of the whole =bytecode=
   test.

4. =run=:: now that the program has been successfully compiled, it is
   run with its standard output and error streams saved to a file.

5. =check-program-output=:: this time it is the output of the program
   which is compared to a reference file, namely the =hello.reference=
   file created earlier. So far this comparison succeeds, because the
   output of the program is identical to the reference file but, as an
   exercise, one may try to modify the reference file to see how this
   causes the failure of this action and of the whole =bytecode= test.

   This action concludes the test of the =ocamlc.byte= compiler. We now
   know that it is able to successfully compile our test program and that
   the resulting executable runs as expected. The four remaining actions
   are going to test the =ocamlc.opt= compiler in a similar but not
   identical way:

6. =setup-ocamlc.opt-build-env=:: this action is the counterpart of
   action 1 for the =ocamlc.opt= compiler.

7. =ocamlc.opt=:: like action 2, this action compiles the test program
   but with the =ocamlc.opt= compiler.

8. =check-ocamlc.opt-output=:: again, this action is similar to
   action 3.

9. =compare-bytecode-programs=:: here we make sure that the generated
   executable is correct, but in a different way than for the
   =ocamlc.byte= compiler. Rather than running it and checking its
   output, we compare it to the one produced in action 2. Such a check
   may seem strange, because what it requires is that =ocamlc.byte= and
   =ocamlc.opt= produce exactly the same binary and not two binaries
   that perform similarly when they are run, but it has proven useful in
   the past and has permitted to detect a subtle bug in the compiler.

** Customizing the default tests

As has been briefly mentioned, the precise behavior of actions (and
thus of tests) may depend on /variables/ whose value can be adjusted in
the =(* TEST ... *)= blocks. In ocamltest, all the values of variables
are strings. Here are a few examples of things that can be achieved just
by defining the appropriate variables. The complete description of the
actions provided by ocamltest and the variables they use will be given
in chapters [[#builtins]] and [[#ocaml-specific]].

*** Passing flags to the compilers

Assume our =hello.ml= example is modified as follows:

#+begin_src
(* TEST *)

open Format

let _ = print_endline "Hello, world!"
#+end_src

As may be verified, this program still passes the default tests. It is
however not as minimal as our previous version, because the =Format=
module is opened but not used. Fortunately, OCaml has a warning to
detect such unused =open= directives, namely warning 33, which is
disabled by default. We could thus add this version of =hello.ml=
to the test suite, not so much to verify that the program compiles and
runs as expected (we verified this already), but rather to make sure
the compiler does indeed trigger the expected warning. Here are the
required steps to achieve this:

1. We slightly modify the test block in =hello.ml=, as follows:
   #+begin_src
   (* TEST
     flags = "-w +33";
   *)
   #+end_src

2. Since we now expect a non-empty output for the compilers, we need to
   store the expected output in a file, namely =hello.compilers.reference=
   beside =hello.ml= and =hello.reference=. To figure out what
   this file shall contain, we can run ocamltest even before it
   has been created. Of course, the action that checks compiler output
   will fail, but in this way we will get the compiler's output
   which we will just have to check (to
   make sure it is what we expect) and to move to the reference file.
   Thus, we do:
   : $ ocamltest hello.ml
   which fails, unsurprisingly, and shows us the paths to the file
   containing the output produced by the compiler and the path to the
   expected reference file. We also see what the compiler produced as
   output but we can double-check that the output is what we expect as a
   reference:
   : $ cat _ocamltest/hello/ocamlc.byte/ocamlc.byte.output
   which shows the warning we expect from the compiler. We can thus move
   this file to the reference file:
   : $ mv _ocamltest/hello/ocamlc.byte/ocamlc.byte.output hello.compilers.reference
   and if we now run ocamltest again, all the tests pass.

Two remarks are due. First, we have used the =flags= variable, to pass
extra flags to all the compilers. There are two other variables one can
use, namely =ocamlc_flags= and =ocamlopt_flags=, to pass flags to the
bytecode or native compilers. Second, in this test all the compilers
have the same output so one reference file is enough for all of them.
There are situations, though, where the compiler's output is
back-end-specific (it depends whether we compile to bytecode or to native
code) or even compiler-specific. ocamltest is clever enough to know how
to deal with such situations, provided that the reference files are
named appropriately. It will indeed first lookup the test source
directory for a compiler-specific reference file, e.g.
=hello.ocamlc.byte.reference=. If no such file exists, a
back-end-specific reference file is searched, e.g.
=hello.ocamlc.reference= for a reference common to both =ocamlc.byte= and
=ocamlc.opt=. If this file does not exist either, ocamltest falls back
to looking for =hello.compilers.reference= as we have seen in this
example, the absence of which meaning that the compiler's output is
expected to be empty.

*** Using an auxiliary module

Let's start with our original =hello.ml= test program and extract the
greeting logic into a distinct =greet.ml= module:

#+begin_src
let greet guest = Printf.printf "Hello, %s!\n" guest
#+end_src

Let's also write an interface, =greet.mli=:

#+begin_src
val greet : string -> unit
#+end_src

Our =hello.ml= test program can then be rewritten as follows:

#+begin_src
(* TEST
  modules = "greet.ml";
*)

let _ = Greet.greet "world"
#+end_src

Provided that the =hello.compilers.reference= file previously used to test
warnings is deleted, running ocamltest on =hello.ml= should work. It
will also be worth looking at the first two lines of the log file generated
while running the test. It says:

#+begin_src
Specified modules: greet.ml hello.ml
Source modules: greet.mli greet.ml hello.ml
#+end_src

The first line shows that the =modules= variable has been taken into
account. On the second line, it can be seen that the =greet.mli= file
appears, right before =greet.ml=. It is ocamltest that has added it,
because it has been recognized as an interface for one of the specified
modules.

To sum up, if a test consists in several modules, it is enough to list
their implementations (except the one of the main test program which is
implicit) in the =modules= variable, in linking order. There is no need
to worry about their interfaces, which will be added automatically by
ocamltest, if they exist.

*** Linking with a library

Assume we want to use the following program to make sure regular
expressions as implemented by the =Str= library work as expected:

#+begin_src
let hello_re = Str.regexp "^Hello, .+!$"

let hello_str = "Hello, world!"

let _ =
  if not (Str.string_match hello_re hello_str 0) then
  begin
     Printf.eprintf "There is a problem!\n";
     exit 2
  end
#+end_src

This test terminates silently if everything goes well and prints a
message on its standard error only if something goes wrong, which means
we won't have anything special to do so that ocamltest checks for an
empty output after the program has run. However, to be able to compile
and link this test, there are several things we need to do so that it
finds the =Str= library it uses. More precisely, we need to add the =-I=
option pointing to the right directory and, at link time, to give the
name of the appropriate library file. To make our life a bit simpler,
ocamltest has a few variables where directories and libraries can be
listed. Once they are there, it is ocamltest which will take care of
adding the =-I= option for each directory and for adding the right
library file depending on whether we are producing bytecode or native
programs. So, here is how the previous program can be annotated so that
it becomes a test:

#+begin_src
(* TEST
  directories += " ${ocamlsrcdir}/otherlibs/str ";
  libraries += " str ";
*)
#+end_src

With these annotations, it becomes possible to run =re.ml= as an
ocamltest test program and, doing so, one may notice that the two tests
pass. There are however a few other things worth pointing out here
regarding the ocamltest DSL. For a start, the notation =${variable}=
inside a string means to replace =variable= by its value, as happens in
many other languages, like the bash shell. Moreover, it is the first
time we meet the ~+=~ operator which concatenates a value to a variable.
More precisely,
: foo += "bar"
is equivalent to
: foo = "${foo}bar"
and not to
: foo = "${foo} bar"
as it may happen in other languages such as makefiles.

In other words, the ~+=~ operator concatenates two strings without
inserting any implicit space between them as e.g. make would do. This is
because in some cases such a behavior is required and could not be
achieved if spaces were implicitly added, whereas with a literal
concatenation it is always possible to include spaces explicitly. This is
exactly what happens in the ocamltest annotation block above, where the
strings added to the =libraries= and =directories= variables are
surrounded by spaces. As should be clear to the reader by now, these
spaces are mandatory. Without them, the added values would be glued to
the last word of the variable and would thus be misinterpreted.

Finally, one may notice that, although ocamltest does make it
possible to link a test program with a library, it does not really make
it easy or convenient to do so. In particular, what if we want to write
several, perhaps many test programs that need to be linked with =Str=?
Will we have to repeat these lines everywhere, thus creating code that
is going to be tedious to maintain? Well, fortunately not. Actually,
ocamltest has a much more elegant way to deal with such issues, namely
/environment modifiers/. As will be explained in chapter [[#concepts]], an
/environment modifier/ is an object that gathers several variable
definitions that can then be included in an ocamltest block at once.
Environment modifiers have to be defined in ocamltest itself and can
then be used with the =include= directive. For instance, the previous
test block is actually written as follows:

#+begin_src
(* TEST
  include str;
*)
#+end_src

*** Testing only on Unix systems

So far, we have been able to fulfill our requirements just by assigning
the right values to variables and relying on the =bytecode= and =native=
tests ocaml runs by default. There are however situations where this is
not enough and where one needs the ability to run other tests. One
example of such a situation is when a test needs to be performed only on
one given operating system, e.g. because it uses a feature which is
present only on that operating system. On an other operating system, the
test should be skipped because it is irrelevant. To illustrate this,
here is how our original =hello.ml= test program should be annotated so
that it is run only on Unix platforms:

#+begin_src
(* TEST
  unix;
  {
    bytecode;
  }
  {
    native;
  }
*)
#+end_src

As can be understood from this example, tests are organised in a tree
nested blocks. Each block begins with a brace and a list of tests and
environment statements that are executed in sequence. Then the block
contains a set of sub-blocks that are executed independently of each
other (i.e. their environments are independent and they are run
regardless of the success or failure of their siblings).
Here for instance, =bytecode= and =native= are sub-tests that will be
run only if the =unix= test passes and will not be started if it fails
or skips.

With this information in mind, it can be seen that the smallest test
block
: (* TEST *)
is actually equivalent to
#+begin_src
(* TEST
  {
    bytecode;
  }
  {
    native;
  }
*)
#+end_src

One common error when designing tests is to believe that a block like
#+begin_src
(* TEST
  unix;
*)
#+end_src
means to execute the =unix= test that verifies that the OS is indeed
Unix and then to execute the default tests. This is actually not the
case. The only situation in which the default tests are considered is
when the test block contains absolutely no test statement.
As soon as there is a test statement, the
default tests are ignored completely and one needs to be totally
explicit about which tests to run. So the correct way to write the
erroneous block above is the use shown at the beginning of this section,
namely:
#+begin_src
(* TEST
  unix;
  {
    bytecode;
  }
  {
    native;
  }
*)
#+end_src

The braces make explicit the scope of variable assignments: an
assignement modifies a variable for the rest of its block and for all
sub-blocks (unless overridden at some point).

For instance, given the following blocks:
#+begin_src
(* TEST
  foo = "abc";
  {
    bar = "def";
    test1;
    {
      baz = "hij";
      subtest1;
    }
    {
      subtest2;
    }
  }
  {
    test2;
  }
*)
#+end_src
- The definition of =foo= is visible in all the tests

- The definition of =bar= is visible in all the tests except =test2=.

- The definition of =baz= is visible only in =subtest1=.

** Other useful tests

This section introduces three tests provided by =ocamltest= and that can
be of particular interest. A brief description of each available test is shown
by the option `-show-tests` of ocamltest. A complete list of available tests
and actions and their detailed descriptions are given in chapters
[[#builtins]] and [[#ocaml-specific]].

*** Testing the top-level: the =toplevel= and =expect= tests

Two tests are provided to make sure that the OCaml top-level behaves as
expected: =toplevel= and =expect=. These tests are similar in that they
both allow testing how the OCaml top-level reacts to some user input,
but they are different in the way one specifies the expected output and
also in what they can test. The =toplevel= test behaves in a spirit
similar to the compiler tests described above, meaning that the expected
output has to be stored in its own, separate file. Since this test
invokes the real OCaml toplevel, it is useful to test advanced features
like the behavior of the toplevel when its input is a file rather than
a terminal, or similar things. In the expect test, on the contrary,
the input and the output it is expected to produce can be written in
the same file, close to each other. However, this test uses the OCaml
toplevel as a library, rather than calling it as an external program.
So this test is actually not testing the complete real OCaml toplevel,
but for testing language features it remains perfectly valid and is
actually what is needed in most of the cases. We thus give below an
example of an expect test and will describe the =toplevel= test in
chapter [[#ocaml-specific]].

So, here is a toy example of an =expect= test:

#+begin_src
(* TEST
  expect;
*)

type point = { x : int; y : int };;
[%%expect{|
type point = { x : int; y : int; }
|}];;
#+end_src

The first line after the test block is the input phrase, while the line
that appears between =[%%expect{|= and =|}];;= is the corresponding
expected output. The =expect= test can also be used to test the output
in presence of the =-principal= command-line flag. In such cases, the
expected output should be written in a =|}, Principal{|= block (to be
improved).

*** The =script= test

It may happen that a needed test is not provided by ocamltest. Of
course, if it turns out that this test would be helpful to test several
source files, then the best solution is to add it to ocamltest itself.
Some tests are however so specific that it is easier to write them as
shell scripts. Such tests can be run by the =script= test, their name
being defined by the =script= variable. In this case, the script is run
in an environment where all the variables defined in ocamltest have been
exported. The script uses its exit status to report its result and can
write a response to a dedicated file to modify its environment or
explain why it failed or skipped, as will be explained in chapter
[[#builtins]]. For the moment, let's see how to use a script to "test" our
original =hello.ml= example. Our annotated program would look as
follows:

#+begin_src
(* TEST
  script = "${test_source_directory}/faketest.sh";
  script;
*)

let _ = print_endline "Hello, world!"
#+end_src

And here is =faketest.sh=, make sure it is executable:

#+begin_src
#!/bin/sh
exit ${TEST_PASS}
#+end_src

This should be enough for the following command to work:
: ocamltest hello.ml

This of course tests nothing and a real test script should actually do
something before returning its result. Let's however see how we can
make the script test fail gracefully:

#+begin_src
#!/bin/sh
echo Why should this pass in the first place > ${ocamltest_response}
exit ${TEST_FAIL}
#+end_src

Running ocamltest on our =hello.ml= program again produces the following
output:
#+begin_src
 ... testing 'hello.ml' with 1 (script) => failed (Why should this pass in the first place)
#+end_src

* Key concepts
  :PROPERTIES:
  :CUSTOM_ID: concepts
  :END:

** Actions, hooks and tests

** Semantics of a test block

** Variables, environments and how they are inherited

** Builtin variables

The list of builtin variables can be obtained by running =ocamltest
-show-variables=.

** Environment modifiers

Environment variables for a test can be set using:
#+begin_src
set VARIABLE_NAME="value";
#+end_src
in the test header (the quotes are mandatory).

On the contrary, you can ensure that an environment variable is not set when
the test runs with:
#+begin_src
unset VARIABLE_NAME;
#+end_src

* Built-in actions and tests
  :PROPERTIES:
  :CUSTOM_ID: builtins
  :END:

The list of builtin and OCaml-specific actions and tests can be obtained by
running =ocamltest -show-actions=.


* OCaml-specific actions and tests
  :PROPERTIES:
  :CUSTOM_ID: ocaml-specific
  :END:

# Things to document (requested by Leo on caml-devel)
# - the syntax of the DSL
# - a clear definition of what "test" means in the context of the DSL
# - a list of the builtin "actions"
# - a list of which "actions" depend on which "variables"
# - what does "include" do?
# - what is the scoping of variables?

# LocalWords: ocamltest OCaml DSL extensibility makefiles

# Local Variables:
# ispell-local-dictionary: "english"
# End: