Subject: Re: CVS commit: pkgsrc/benchmarks/bytebench
To: Todd Vierling <tv@duh.org>
From: Urban Boquist <urban@boquist.net>
List: tech-pkg
Date: 11/02/2005 08:16:39
>>>>> Todd Vierling writes:

Todd> While there, use fsync(2) on all hosts.  Without this, the
Todd> fstime benchmark will be disproportionately biased toward OS's
Todd> and hardware that buffer larger amounts of data in memory before
Todd> [background] syncing.

>> the second part of this change seems inappropriate, IMHO. The
>> benchmark may be biased, but that is not up to pkgsrc to
>> decide. Pkgsrc is just supposed to be a way to port software, it
>> should not change the behavior of an old and well defined
>> benchmark.

Todd> Actually, I just disproved fstime's methodology.  Without using
Todd> fsync(2), the numbers are *meaningless*.

Sure, I don't dispute that. I'm just saying that you should take that
up with the authors of bytebench. You are not supposed to redesign a
well known benchmark on your own like this.

I can easily see how this will lead to bad press for NetBSD when
someone runs this and compares the numbers with either some published
bytebench results or some friend running another OS. At best it will
lead to unnecessary confusion.

Best regards,

        -- Urban

Todd> Testing filesystem speed accurately requires actually writing to
Todd> the disk -- and if you don't fsync(2), you can very well end up
Todd> with a "write speed" of one or two orders of magnitude higher
Todd> than a disk is physically capable of performing.

Todd> It should wither get fsync(2), or be removed from the battery of
Todd> tests.  We should not be providing a bogus test in a supplied
Todd> benchmark program.