Subject: Re[2]: benchmark
To: Dries Schellekens <gwyllion@ulyssis.org>
From: Igor Shmukler <shmukler@mail.ru>
List: tech-perform
Date: 01/25/2005 19:31:16
Dries,
If you read this thread of the one from FreeBSD performance carefully, you will notice following.
Nobody EVER argued with the paper or its content.
Actually, as far as I remember original article on OSNews had a normal title too - NetBSD scales 
better (or something like this).
So to answer your questions - YES.
BTW, there are problems with the benchmark suite. We used for an internal project but after 
changing some of the benchmarks, otherwise results MAY not be correct.
I contacted author to provide details (back when this was originally published), but he did not 
follow up. Perhaps spam filter swollowed my email.
The acticle is NOT a rant at all. It simply an analisys of a scalability test-suite.
Igor.

>>  About flaming. I am not a fan of publically humiliating anyone.
>> It's a wrong thing to do. In fact when the person who started this
>> sent  an email to FreeBSD performance list I merely pointed out that
>> microbenmark != performance. It does not mean that NetBSD performs
>> worse than FreeBSD. In fact it is possible that NetBSD is better,
>> although I want to see a proof before such a statement could be made.
> 
> Did you read the conclusions of the paper?
> 
>> Microbenchmarks are not always the best indicators to make judgments on 
> the overall performance of one operating system over another. However, 
> they are useful to infer an understanding of the architectural decisions 
> that go into building an operating system. For many applications, the 
> results presented in the paper may never affect performance. For others, 
> the scalability of the operating system may simply not permit the 
> application to run suitably.&quot;
> 
> and
> 
> > Although NetBSD 2.0 has outperformed FreeBSD 5.3 in most of the 
> benchmarks presented here, FreeBSD 5.3 has made significant developments 
> with its symmetric multiprocessor (SMP) architecture, particularly in 
> the area of scalability with fine-grained locking. NetBSD 2.0 continues 
> to use a single lock to serialize access to kernel mode. Additionally, 
> the performance of the thread implementation on multiprocessor systems, 
> where thread concurrency can be achieved, would be worth investigating. 
> Benchmarks for these areas are the objective of future research.&quot;
> 
> 
> Cheers,
> 
> Dries
>