Re: atf for libcurses

[Antti Kantee <pooka%netbsd.org@localhost>]

On Thu Nov 04 2010 at 12:36:34 +1030, Brett Lymn wrote:
> On Wed, Nov 03, 2010 at 01:00:08PM +0000, Antti Kantee wrote:
> > 
> > Awesome!  Soon I'll have the courage to touch curses and be sure
> > I didn't break anything.
> > 
> 
> Well, that is the whole idea of doing this :)

So you understand the point of automated tests, but based on some
conversations I've had not everybody does.

> > Apologies if I missed it, but I didn't find how to create the
> > expected output files.  The ones I did look at looked, hmm, cryptic,
> > so if there some tool to create an output check file which verifies
> > that "hello" is printed in exactly the right place on the screen?
> > 
> 
> Sorry for arm waving over that.  This is probably the hardest bit at
> the moment.  What I don't want to do is automate capturing current
> output because there may be lurking bugs and we would just enshrine
> them into a test which I think is bad.

Can you expand on what kind of bugs you are worried about?

> What I have done is set up a custom terminfo database for a terminal
> called "atf", most of the terminfo output sequences are mapped to
> their names.  If you look in the source files at the atf.terminfo file
> you will see what I mean.  When curses uses the terminfo capabilities
> they will be emitted as their names.  So the output stream should be
> somewhat readable.

oic

> What I have been doing so far is setting the check file to /dev/null
> and running the director in verbose mode (-v) which will tell you what
> characters in the output stream were mismatched.  Having a look and
> think about the output stream and if I believe it correct then putting
> that sequence into a check file.

Doesn't that capture the bugs you were worried about in the same way?
I seriously doubt people will read the result especially for more
complicated tests where the terminfo line noise is several lines or
even pages long.  Massive "golden" files might be common in $application
test cases.

> > Also, can this scheme be extended to curses applications (say, vi)
> > and even further to similar applications which don't necessarily
> > use curses?
> > 
> 
> Yes, I think this is doable.  In fact I think it is almost there.  I
> have had a bit of a rethink about how I do the output validation.  At
> the moment the director looks for a set of function calls that it has
> been told will result in output and expects an extra argument to that
> call which is the check file.  I think this is a bit limiting now and
> that I should introduce a new keyword, something like "validate" or
> "verify_output" that will use the check file and perform the stream
> comparisons.  If I do this then it would be a matter of doing a new
> slave that can exec a program and drive it using the "input"
> directives.  Definitely worth thinking about.

Ok, sounds easy enough.  IIRC someone noticed that nvi developed a
regression where it left the cursor at the wrong place and wanted to
write a test for it.  So in your scheme it would be a matter of sending
the right commands to vi with "input" and figuring out what the expected
output is.

> > (and finally, do you have a script which ascends in nethack ?)
> 
> Given the last time I played rogue my dog bit me and I went blind from
> drinking some random potion... I think not.

It's easier if you're blind since there are less variables ;)