ATF-devel archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Renewed data formats
Hi,
I've finished renewing the parsers and documenting the different data
formats used in ATF. This is still not completely ready as the code
needs some cleanups (as always) and it is lacking a lot of unit
tests, but the major goal for 0.2 is almost done.
I'm pasting below the new manual page that details the contents of
the different data formats. It may still be too sketchy but should
have enough information to give you an idea of how they are.
Should you have any suggestion for it, feel free to raise it.
Otherwise these formats will stay as they are now until there is a
real need to change them in the future (to lessen gratuitous
incompatibilities).
Thank you!
----- cut here -----
ATF-FORMATS(5) BSD File Formats Manual ATF-FORMATS(5)
NAME
atf-formats -- machine-parseable data formats used by ATF
DESCRIPTION
This manual page describes the multiple data formats used in
ATF. These
formats affect configuration files, control files and any data
that is
externalized or internalized by the tools.
Data files are always organized as follows:
Header1: Value1 \
... | head
HeaderN: ValueN /
mandatory blank line
Free-form text contents \
... | body
... /
A file must always contain a `Content-Type' header and must
always sepa-
rate that header from the body with a blank line, even if the
body is
empty.
The `Content-Type' is always of the form:
Content-Type: application/X-atf-<subtype>; version="<version>"
where `subtype' indicates the specific file format and
`version' its for-
mat version. This header must be the first one of the file.
The main purpose of the `Content-Type' header, aside from
determining the
format used in the file, is to allow future changes to a given
format.
Whenever an incompatible change is made, the version is bumped
by one.
By keeping the header in the first line, future versions may
even remove
the need for such a header -- e.g. if some format was replaced
by XML
files, which have their own mandatory header.
The rest of this document details the different format types.
Format: application/X-atf-atffile, version: 1
Atffiles are logically divided into three sections:
Test programs: the list of test programs that define the test
suite
described by the Atffile.
Meta-data properties: these define some constant values
applicable to
all the test programs defined in the file. In some sense they define
the properties that describe the test suite.
Configuration variables: defaults for configuration variables
that
can be overridden through configuration files or the command line.
The grammar for Atffiles is the following:
DATA ::= ( ( CONF | PROP | TP )? COMMENT? NEWLINE )* EOF
CONF ::= 'conf:' STRING EQUAL STRING
PROP ::= 'prop:' STRING EQUAL STRING
TP ::= TPFILE | TPGLOB
TPFILE ::= 'tp: ' STRING
TPGLOB ::= 'tp-glob: ' STRING
The meaning of the constructions above is:
CONF Definition of a configuration variable.
PROP Definition of a meta-data property.
TPFILE Addition of a test program into the test suite. The
string is
taken literally as the program's name, and this program must
exist.
TPGLOB Addition of multiple test programs into the test
suite. The
string is taken as a glob pattern, which may have or not have
any matches in the current directory.
An example:
prop: test-suite = utilities
conf: unprivileged-user = nobody
tp: t_cp
tp: t_mv
tp: t_df
tp-glob: t_dir_*
Format: application/X-atf-config, version: 1
Configuration files are very simple: they only contain a list
of variable
name/variable value pairs. Their grammar is:
DATA ::= ( VAR? COMMENT? NEWLINE )* EOF
VAR ::= STRING EQUAL STRING
COMMENT ::= HASH STRING
An example:
# This is the system-wide configuration file for ATF.
# The above and this line are comments placed on their own line.
var1 = this is a variable value
var2 = this is another one # Optional comment at the end.
Format: application/X-atf-tcs, version: 1
The `application/X-atf-tcs' format is used to describe the
results of a
collection of test cases; in other words, it represents the
output of a
test program. Unfortunately, it is not easy to control, from
inside a
test program, what it prints to both its standard output and
standard
error streams. This is specially the case of test programs
written in
the POSIX shell language, because they are constantly executing
external
tools that may print unexpected messages at all times. Due to
this, ATF
imposes no restrictions on what a test program can send to
these two
channels; in fact, they are encouraged to print as much useful
informa-
tion as possible to aid in the debugging of test failures.
Because we have no control over the two standard streams, the
`application/X-atf-tcs' format describes the structure of a
third stream,
known as the results output, that test programs must generate.
(Note
that test programs send, by default, the results output to the
standard
output; use their -r flag to change this whenever you need to
parse the
data.) This stream is decoupled from the other two and has the
following
grammar:
DATA ::= TCS-COUNT TC-STANZA* EOF
TCS-COUNT ::= 'tcs-count' COLON POSITIVE-NUMBER NEWLINE
TC-STANZA ::= TC-START TC-END
TC-START ::= 'tc-start' COLON STRING NEWLINE
TC-END ::= 'tc-end' COLON STRING COMMA TCR NEWLINE
TCR ::= 'passed' | ('failed' | 'skipped') COMMA STRING
The meaning of the constructions above is:
TCS-COUNT Indicates the number of test cases that will be
executed.
There will be this exact amount of `TC-STANZA' constructions
following it.
TC-START Indicates the beginning of a test case. This is
accompanied
by the test case's name.
TC-END Indicates the completion of a test case. This is
accompa-
nied by the test case's name, its result and the reason
associated with this result (if applicable).
There are multiple reasons behind this design:
The reader of this format must be able to show real-time
progress to
the user as the test cases are processed. Therefore, the `TC-START'
construction tells the reader when a test case has started to process
data.
The reader of this format has to be able to provide useful
statistics
to the user without having to wait for the end of the file. Hence,
the existence of the `TCS-COUNT' construction located at the begin-
ning of the file.
Text-based tools have to be able to easily look for the
results of a
given test case. This is why the `TC-END' construction duplicate the
test case name already provided in `TC-START'.
An example:
tcs-count: 2
tc-start: add
tc-end: add, passed
tc-start: subtract
tc-end: subtract, failed, Calculated an unexpected value
Going back to the standard output and standard error streams,
the reader
has to be able to match the messages in those two streams to
the test
cases they belong to. To do this, these two streams must print
a magic
string that separates the output of test cases from each other,
which is
enough to synchronize their contents with the results output.
This
string is `__atf_tc_separator__' and it must printed on a line
of its
own. The last test case should not be followed by this line
because the
end of file marker takes its role.
Format: application/X-atf-tps, version: 1
The `application/X-atf-tps' format multiplexes the standard
output, stan-
dard error and results output streams from multiple test
programs into a
single data file. This format is used by atf-run(1) to report
the execu-
tion of several test programs and is later parsed by atf-report
(1) to
inform the user of this process. It has the following grammar:
DATA ::= TPS-COUNT TP-STANZA* EOF
TPS-COUNT ::= 'tps-count' COLON POSITIVE-NUMBER NEWLINE
TP-STANZA ::= TP-START TC-STANZA* TC-END
TP-START ::= 'tp-start' COLON STRING COMMA POSITIVE-NUMBER NEWLINE
TP-END ::= 'tc-end' COLON STRING (COMMA STRING)?
TC-STANZA ::= TC-START (TC-SO | TC-SE)* TC-END
TC-START ::= 'tc-start' COLON STRING NEWLINE
TC-SO ::= 'tc-so' COLON STRING NEWLINE
TC-SE ::= 'tc-se' COLON STRING NEWLINE
TC-END ::= 'tc-end' COLON STRING COMMA TCR NEWLINE
TCR ::= 'passed' | ('failed' | 'skipped') COMMA STRING
The meaning of the constructions above is:
TPS-COUNT Indicates the number of test programs that will be
executed.
There will be this exact amount of `TP-STANZA' constructions
following it.
TP-START Indicates the beginning of a test program. This
includes
the program's name and the amount of test cases that will
follow.
TP-END Indicates the completion of a test program. This is fol-
lowed by the program's name and, if the program ended prema-
turely, an error message indicating the reason of its fail-
ure. A successful execution of a test program (regardless
of the status of its test cases) must not be accompanied by
any reason.
TC-START Indicates the beginning of a test case. This is
accompanied
by the test case's name.
TC-SO Contains a text line sent to the standard output stream
dur-
ing the execution of the test case.
TC-SE Contains a text line sent to the standard error stream
dur-
ing the execution of the test case.
TC-END Indicates the completion of a test case. This is
accompa-
nied by the test case's name, its result and the reason
associated with this result (if applicable).
An example:
tps-count: 2
tp-start: calculator, 2
tc-start: add
tc-end: add, passed
tc-start: subtract
tc-so: 3-2 expected to return 1 but got 0
tc-end: subtract, failed, Calculated an unexpected value
tp-end: calculator
tp-start: files, 1
tc-start: copy
tc-se: could not find the cp(1) utility
tc-end: copy, skipped
tp-end: files
SEE ALSO
atf(1)
BSD September 6, 2007 BSD
----- cut here -----
--
Julio M. Merino Vidal <jmmv84%gmail.com@localhost>
Home |
Main Index |
Thread Index |
Old Index