Re: Initial Kyua import done

[Marc Balmer <marc%msys.ch@localhost>]

> 
> I am very happy to announce that the initial import of Kyua[1] into the
> NetBSD source tree[2] is complete!  I am sure there are some rough edges
> and here is where you come into play.
> 
> First of all, let me clarify that all the integration changes are gated
> by the MKKYUA build setting, which still defaults to no.  Unless you
> explicitly set MKKYUA=yes in /etc/mk.conf or in the build.sh command
> line, you will /not/ get a system with Kyua.  However, once you set that
> flag, you will transition to the full new setup:

Why don't you just enable it by default?  I don't see the point of making it 
optional, it's new stuff that gets added, it does not break existing stuff, 
right?

> 
> * Kyua will be installed as /usr/bin/kyua.  See kyua(1) to get started.
> 
> * The old atf-run and atf-report tools will become compatibility
>  wrappers.  These should be a reasonable drop-in replacement for most
>  use cases, but they are probably not perfect.
> 
> * Your /usr/tests tree will be populated by a bunch of Kyuafiles.
> 
> So what's new compared to the ATF tools?  Here are some highlights:
> 
> * Ability to write and run ATF-less tests.  It has been a common desire
>  around here to develop test programs that do not rely on the ATF
>  libraries.  With this change, this becomes possible.  See kyuafile(5)
>  for details.
> 
> * Direct HTML report generation.  There is no need to set up a complex
>  XML toolchain any more to convert the ATF test reports into HTML
>  pages.  Kyua can generate plain HTML directly, so it will be feasible
>  to serve such content straight from NetBSD's built-in httpd.
> 
> * Historical data.  As seen in the various test beds that have appeared
>  around ATF, there is a desire to maintain historical data of the test
>  results.  Kyua does that natively, by recording the results of the
>  execution in a SQLite database.  Reports can later be extracted from
>  this database.  There is still a lot of room for improvement here.
> 
> * More flexible metadata and configuration.  While this does not provide
>  a real advantage today, as soon as the old atf-run and atf-report
>  tools are gone we can trivially fix some long-standing issues
>  (e.g. the inability to customize test deadlines).
> 
> * Less complexity during test case execution.  As you may have noticed
>  over the years, the code in atf-run to capture the output of tests and
>  deal with interrupts is not particularly robust.  There have been
>  several problems in this area, and I'm not convinced that they are all
>  fixed.  The new code works in a different manner and has been more
>  carefully thought around these edges.
> 
> * Independent testers.  The code that implements the isolation of test
>  cases and their controlled execution has been split into a set of
>  "testers" that live in /usr/libexec/kyua-*-tester.  These tools
>  provide scriptable interfaces to interact with tests, with the idea
>  that the kyua(1) frontend should end up being pretty straightforward.
>  Should you want to write your own trivial script to run tests without
>  kyua(1), you could pretty easily do that by interacting with the
>  testers directly.
> 
> How can you help?  Easy.  Just rebuild your system with MKKYUA=yes, read
> through kyua(1), start using it to stress-test your system and report
> any problems[3] you may encounter!
> 
> My immediate next steps include addressing your feedback and working
> with our major test runners to add support to their systems to use the
> new tools (for example, change anita to support running tests with
> kyua(1)).
> 
> Enjoy and thanks for reading.
> 
> 1: http://wiki.NetBSD.org/kyua/
> 2: http://wiki.NetBSD.org/kyua/import/
> 3: https://code.google.com/p/kyua/issues/list for kyua-specific bugs
>   and send-pr for NetBSD-specific bugs.
> 
> -- 
> Julio Merino / @jmmv


Freundliche Grüsse,
micro systems

Marc Balmer

--
Marc Balmer
micro systems, Wiesendamm 2a, Postfach, 4019 Basel
fon +41 61 383 05 10, fax +41 61 383 05 12, http://www.msys.ch/