Subject: Format of test programs' output
To: None <atf-devel@NetBSD.org>
From: Julio M. Merino Vidal <firstname.lastname@example.org>
Date: 08/25/2007 16:03:22
I have started cleaning up the current mess of data formats and their
parsing. I would like to get some data formats that can remain
stable for several versions and be easy to process in multiple
situations. The former is, IMO, a pre-requisite for integrating ATF
into NetBSD; otherwise frequent changes in the format will cause pain
to users even if the differences are handled gracefully.
So I have started with the format that test programs print during
execution. Remember that, at the moment, a test program is just a
collection of test cases that presents a consistent CLI to the user.
There are requisites for this format:
- As these are programs, we cannot restrict them from printing stuff
to their stdout and stderr, hence we must be able to capture these
two as part of the test program's execution and not be disturbed by
data that should not be there (e.g. printed by the framework itself).
- Test cases can take a while to execute, yet we want to keep the
user informed of progress. We must print some stanzas before the
test case is executed and after this happens.
- The results are something separate from stdout and stderr. The
reader should be able to parse this alone and not be disturbed with
other output from the program. In other words, the results cannot
be printed to the stdout.
Here is the format I have in my mind:
tcs-count: <number of test cases that follow>
tc-start: <name of 1st test case>
tc-end: <name of 1st test case>, passed|failed|skipped[, reason]
tc-start: <name of 2nd test case>
tc-end: <name of 2nd test case>, passed|failed|skipped[, reason]
tc-start: <name of nth test case>
tc-end: <name of nth test case>, passed|failed|skipped[, reason]
<end of file>
- Reason is only present for failed and skipped test cases.
- The test case name is repeated on the tc-end line so that it is easy
to grep for specific test case results, without having to filter it
through atf-run nor atf-report. This can be convenient in multiple
situations. Simply doing something like "grep '^tc-end: foo, passed'
should be enough to ensure that the test case 'foo' passed, for
example. This is not possible with the current format, which prints
stuff on different lines.
- The total number of test cases is printed at the beginning so that
the reader can know how many test cases are left during execution
and provide better status messages to the user.
As regards stdout and stderr, it is important to separate the output
generated by each test case (if any) for reporting purposes. E.g.
when a test case fails, we want to show all its output to the user
(read, developer) for diagnosis purposes. I haven't found any way to
synchronize different streams with each other... so I force test
programs to print "__atf_tcs_separator__" on a line of its own when
they have finished.
This is, of course, all handled by the ATF libraries so test cases
don't have to implement any of the above on their own.
Do you see any important data/requirement missing from the above?
Julio M. Merino Vidal <email@example.com>