On Tue, 25 Mar 1997, Allen Briggs wrote:

[Small discussion about what could affect Benchmarking]

> Not only that, but just adding/removing "options DIAGNOSTIC" (or many
> other options--GENERIC is supposed to work on all systems, but not
> necessarily be fast...  ;-) for a given kernel could make a big difference.
> That's the sort of thing that concerned me.

Indeed, kernel configuration can affect performance. I'm just not sure 
how best to handle this :(  I'd be glad to receive suggestions from 
anyone out there who has a good idea...

I did the experiment of running the Bencharks on my IIvx simply hcanging 
the kernel to one built with the same source but without DIAGNOSTIC. The 
"Pipe-based Context Switching Test" was better by about 10% and the 
others were same or better by less than 2%. I'm not sure how much the 
other configs would impact this.

The benchmarks should be taken with a grain of salt for many reasons to 
start with and I guess the fact that kernel options were not taken into 
account makes them even more "ballpark figures". Nevertheless, unless the 
other options have even more effect than the DIAGNOSTIC, the results can 
give a relatively fair estimate of performance, the difference between 
machines beeing usually quite large.

I personally am more afraid that people be mislead about the speed of 
systems by the difference between the particular benchmark results and 
the "real-life-apps" performance. This lies with the quality of the 
Benchmark suite itself which I hope to be good but have not verified. It 
was chosen because it was free and seemed used by many other people on 
the Net.

Once again, any suggestions/comments about the Benchmarking that I put up 
(http://www.cs.mcgill.ca/~magus/unixbench.html) is welcome.

-----------------------------------------------------------------
"Because user errors often produce unpredictable results, the user
should try to avoid them."

IBM MVS/XA System Programming Library.