Announcing Tezt

Tezt (pronounced /tɛzti/) is a test framework for OCaml that has been developed and used at Nomadic Labs to test Octez, an OCaml implementation of the Tezos blockchain. It has become quite mature and we feel it would benefit the OCaml community at large, so we are releasing it publicly as a standalone product.

Use Cases

Tezt is well suited for:

• unit tests;
• integration tests;
• regression tests.

This allows you to seamlessly use the same framework for most of your tests.

Features

General features:

• easy and flexible test selection from the command-line;
• run tests in parallel in separate processes;
• automatically split tests into well-balanced batches for running in separate CI jobs;
• various reporting options, including JUnit for CI integration;
• write assertions (equalities, inequalities, existence in lists, etc.) that result in nice error messages when they fail.

For integration tests:

• run external processes, automatically logging their output and terminating them at the end of a test;
• declare temporary files that are automatically removed at the end of a test;
• external processes can be run on distant runners through SSH;
• a JSON module is provided to test REST APIs.

For regression tests:

• capture some strings (e.g. output from external processes);
• apply regular-expression-based substitutions to make those outputs deterministic;
• compare with previous captured outputs.

Additionally, substantial effort was put into the user experience. Many small details that may not seem like much individually, but add up to significantly improve everyday use. These include:

• colorful logs: for instance, each external process gets its own color;
• by default, logs are quiet until an error occurs, in which case previous log messages are displayed to get some context;
• an option allows you to see all commands that are used to run external processes, making it easy for you to copy-paste them and reproduce;
• if one of your tests is flaky (i.e. randomly fails), an option allows you to run it in a loop to more easily reproduce the error;
• JSON error messages show the location of ill-typed JSON value;
• all functions are clearly documented.

A Simple Test

Create a dune file containing:

(test (name main) (libraries tezt))

Now create a file named main.ml. To add a test, register it with Test.register:

open Tezt
open Tezt.Base

let () =
  Test.register ~__FILE__
    ~title: "example test"
    ~tags: ["example"; "addition"]
  @@ fun () ->
  if 1 + 1 <> 2 then Test.fail "1 + 1 is not 2";
  unit

let () =
  (* Call this after you are done registering tests. *)
  Test.run ()

Let’s see what is going on here.

• ~__FILE__ tells Tezt which source file is defining this test. You can tell Tezt to run all tests that are registered in a given source file with --file (or -f). Here, that would be --file main.ml.
• ~title should be a short descriptive title for your test. You can tell Tezt to run this particular test with --test 'example test'.
• ~tags is a list of identifiers that can be used to select a subset of your tests from the command-line. For instance, you can run all tests that have tag addition simply by adding addition on the command-line. You can also specify negative tags; for instance, you can have a tag slow for tests that take longer to run, and exclude them with /slow.
• The last argument is a function that implements the test.
• Test.fail raises an exception, similar to Failure. It stops the test. Tezt then cleans up if there is something to clean, and reports the error.
• unit is Lwt.return_unit. All tests are written in the Lwt monad.
• Test.run tells Tezt that all tests have been registered. Tezt takes it from here, handling command-line options and running tests that it should run.

To run your test, run either:

dune runtest

or:

dune exec main.exe

You can get a list of command-line options with:

dune exec main.exe -- --help

In particular, you can try the --list option, which gives you the list of registered tests and their tags. Don’t forget to use -- to separate Dune arguments and Tezt arguments:

dune exec main.exe -- --list

Some basic options that you may be interested in are:

• --list to get the list of registered tests;
• --test TITLE (or -t TITLE) to select a test from its title;
• --file FILE.ml (or -f FILE.ml) to select tests from their source file;
• --verbose (or -v), --info (or -i) to control log verbosity;
• --log-file FILE to store verbose logs in FILE;
• --keep-going (or -k) to continue with remaining tests even if a test fails;
• --jobs N (or -j N) to run up to N tests in parallel.

A Simple Integration Test

An integration test is just a test that runs external processes. It often also manipulates temporary files. Here is a simple example:

open Tezt
open Tezt.Base

let () =
  Test.register ~__FILE__
    ~title: "example integration test"
    ~tags: ["example"; "cat"]
  @@ fun () ->
  let filename = Temp.file "test.txt" in
  with_open_out filename (fun ch ->
    output_string ch "test" ^ string_of_int (Random.int 1000));
  let* output = Process.run_and_read_stdout "cat" [ filename ] in
  if output =~! rex "^test\\d{1,4}$" then
    Test.fail "got %S instead of the expected output" output;
  unit

let () = Test.run ()

Let’s see what is going on here. This tests uses three functions from the Base module of Tezt:

• Temp.file "test.txt" returns a filename of the form /tmp/tezt-1234/1/test.txt. This file will be automatically removed at the end of the test unless you use --keep-temp, or --delete-temp-if-success and the test fails.

• with_open_out opens a file for writing, gives you the out_channel, and ensures the file is closed after.

• let* is Lwt.bind.

Then it uses the Process module to execute cat on the temporary file and check its output. The Process module will log all output from cat in verbose mode. This is particularly convenient for debugging.

Finally, this test uses =~!, which means: check that the left-hand-side does not match the regular expression at the right-hand-side. Regular expressions are often convenient to use with integration tests, so some easy-to-use regular expression operators are provided by Tezt.Base.

You can run this test like any other test. One option that may be useful is --commands (or -c): it tells Tezt to print commands that it runs. It quotes them using shell syntax so that you can copy-paste them into your terminal to easily reproduce.

A Simple Regression Test

Regression tests are tests that produce a string which should never change. This string is stored in a file that you can commit in your repository. When you run your test, the string that it produces will be compared with the contents of this file, and the test will fail if it differs.

Regression tests are in particular useful to check that the output of external processes stay unchanged, so they are often used in integration tests. But they can also be used for unit tests.

Here is an example regression test:

open Tezt
open Tezt.Base

let () =
  Regression.register_test ~__FILE__
    ~title: "example regression test"
    ~tags: ["example"]
    ~output_file: "example.txt"
  @@ fun () ->
  let* output =
    Process.run ~hooks: Regression.hooks
      "git" [ "--help" ]
  in
  Regression.capture (string_of_int (1 + 1));
  unit

let () = Test.run ()

Let’s see what is going on here.

• This test is registered with Regression.register_test instead of Test.register. This tells the test that it shall produce a file with the captured regression output.
• output_file is the name of the file to produce. It is prefixed with tezt/_regressions/, so here the output file is tezt/_regressions/example.txt. You can override this prefix with --regression-dir.
• We use ~hooks to tell the Process module to capture the output of the process in the regression output file.
• We also manually capture the result of a computation.

You need to run the test once with --reset-regressions to generate the output file:

dune exec main.exe -- regression --reset-regressions

Here we also specify the regression tag on the command-line. This is optional but this tells Tezt to only run tests which have tag regression. This tag is automatically added by Regression.register_test.

Next time you run your tests, for instance with dune exec main.exe, the output of the test will be compared with tezt/_regressions/example.txt. If it differs, the test will fail.

Regression tests are convenient because they are fast to write. However, they do have some drawbacks:

• the expected output is not in the source code of the test itself (an alternative for this is to use ppx_expect);

• if the output contains non-deterministic values, such as time data or random values, you first need to replace them with fixed values (this can be done with custom ~hooks).

CI Integration

Our tests run in GitLab’s CI for all merge requests. This prompted us to make Tezt well integrated with the CI. Here are some of the features that were built with that in mind.

• GitLab’s job interface has a limit for how many lines of output can be shown for a given job. By default Tezt only shows detailed logs in case of a failure, and only the most recent lines (the exact amount is a command-line option). This means that we never reach GitLab’s limit. We can still see the full logs if we want: we use the --log-file option and store the log file as an artifact.

• Tezt can generate record files that store the time each test took to run. It can then use those records to automatically split tests into a partition where each subset takes roughly the same amount of time. And you can tell Tezt to run only one of this subset in each CI job. The result is that you can easily split your tests in well-balanced parallel jobs in the CI.

• Tezt can generate JUnit reports. JUnit is supported by GitLab. This gives the ability to show a summary of test results in the merge request interface.

• Because Tezt can run external processes using SSH, you could have the CI runner be the puppetmaster for a bigger cluster of machines. We don’t actually do that, but we could.

Note that outside of the JUnit format, none of these features assume that your CI runs on GitLab, so they should port rather easily to other CIs.

And More

Tezt has a rather long list of command-line options to configure its behavior. Run your executable with --help to see it, or read Tezt’s cli.ml source file. We already had a look at some of these options in section A Basic Test and we already mentioned some others. But there are too many to list here, as Tezt, being heavily used, has aggregated many small yet useful features over the time.

Conclusion

Tezt development started more than two years ago. At the time, we had two frameworks for integration tests. Flextesa, which was written in OCaml but failed to convince all developers (possibly because it focused on interactive tests), and a Python-based framework. Most integration tests were thus written in Python. As primarily OCaml developers, we were eager to write our tests in OCaml.

Tezt thus began with a focus on integration tests and was developed with user experience, simplicity of implementation and ease of use in mind. It quickly convinced a few early adopters at Nomadic Labs, and expanded from there. Now, no new test is written in Python, all new integration tests are written using Tezt.

We eventually realized that Tezt could also be used for unit tests. We did not actually transition to Tezt for unit tests though; most unit tests are written using Alcotest, a test library for OCaml. But some developers are starting to voice preference for Tezt for unit testing, either because they like its focus on user experience, or because they like its CI integration capabilities like auto-balancing, or just because they would prefer to only have one framework. Tezt provides most of Alcotest’s features, except (for now) integration with QCheck, a library for property-based testing. It should, however, not be very hard to integrate QCheck with Tezt, and we may decide to make the jump one day.

It’s now clear to us that Tezt is a success with Octez developers, and we see no reason to keep it for ourselves. Version 1.0.0 was already released on opam a while ago, but with no announcement — it was mainly so that we could use it ourselves on other Tezos-related projects. We just released version 2.0.0 on opam: run opam install tezt and start tezting!

You can find the API documentation here.