Initial Kyua import done

[Julio Merino <julio%meroh.net@localhost>]

Hello all,

I am very happy to announce that the initial import of Kyua[1] into the
NetBSD source tree[2] is complete!  I am sure there are some rough edges
and here is where you come into play.

First of all, let me clarify that all the integration changes are gated
by the MKKYUA build setting, which still defaults to no.  Unless you
explicitly set MKKYUA=yes in /etc/mk.conf or in the build.sh command
line, you will /not/ get a system with Kyua.  However, once you set that
flag, you will transition to the full new setup:

* Kyua will be installed as /usr/bin/kyua.  See kyua(1) to get started.

* The old atf-run and atf-report tools will become compatibility
  wrappers.  These should be a reasonable drop-in replacement for most
  use cases, but they are probably not perfect.

* Your /usr/tests tree will be populated by a bunch of Kyuafiles.

So what's new compared to the ATF tools?  Here are some highlights:

* Ability to write and run ATF-less tests.  It has been a common desire
  around here to develop test programs that do not rely on the ATF
  libraries.  With this change, this becomes possible.  See kyuafile(5)
  for details.

* Direct HTML report generation.  There is no need to set up a complex
  XML toolchain any more to convert the ATF test reports into HTML
  pages.  Kyua can generate plain HTML directly, so it will be feasible
  to serve such content straight from NetBSD's built-in httpd.

* Historical data.  As seen in the various test beds that have appeared
  around ATF, there is a desire to maintain historical data of the test
  results.  Kyua does that natively, by recording the results of the
  execution in a SQLite database.  Reports can later be extracted from
  this database.  There is still a lot of room for improvement here.

* More flexible metadata and configuration.  While this does not provide
  a real advantage today, as soon as the old atf-run and atf-report
  tools are gone we can trivially fix some long-standing issues
  (e.g. the inability to customize test deadlines).

* Less complexity during test case execution.  As you may have noticed
  over the years, the code in atf-run to capture the output of tests and
  deal with interrupts is not particularly robust.  There have been
  several problems in this area, and I'm not convinced that they are all
  fixed.  The new code works in a different manner and has been more
  carefully thought around these edges.

* Independent testers.  The code that implements the isolation of test
  cases and their controlled execution has been split into a set of
  "testers" that live in /usr/libexec/kyua-*-tester.  These tools
  provide scriptable interfaces to interact with tests, with the idea
  that the kyua(1) frontend should end up being pretty straightforward.
  Should you want to write your own trivial script to run tests without
  kyua(1), you could pretty easily do that by interacting with the
  testers directly.

How can you help?  Easy.  Just rebuild your system with MKKYUA=yes, read
through kyua(1), start using it to stress-test your system and report
any problems[3] you may encounter!

My immediate next steps include addressing your feedback and working
with our major test runners to add support to their systems to use the
new tools (for example, change anita to support running tests with
kyua(1)).

Enjoy and thanks for reading.

1: http://wiki.NetBSD.org/kyua/
2: http://wiki.NetBSD.org/kyua/import/
3: https://code.google.com/p/kyua/issues/list for kyua-specific bugs
   and send-pr for NetBSD-specific bugs.

-- 
Julio Merino / @jmmv