kern/45734: performance regression for sequential/random creation of files, as well as stat(2) (ffs fs)

[jym%NetBSD.org@localhost]

>Number:         45734
>Category:       kern
>Synopsis:       performance regression for sequential/random creation of 
>files, as well as stat(2) (ffs fs)
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Dec 22 23:30:00 +0000 2011
>Originator:     Jean-Yves Migeon
>Release:        -current
>Organization:
TNF
>Environment:
NetBSD paris 5.99.53 NetBSD 5.99.53 (GENERIC) #1: Thu Dec  1 18:48:00 CET 2011  
jym@paris:/home/jym/cvs/obj/sys/arch/amd64/compile/GENERIC amd64

>Description:
I observed a performance regression for sequential/randomized creation of 
files, as well as stat(2)ing them, between a GENERIC kernel from 2011-03-01 and 
2011-12-18.

These measurements were all done with bonnie++ (benchmarks/bonnie++), for a ffs 
mounted with -o log, and the following command:

bonnie++ -n 32 -r 4096 -x 3 -u nobody

(32 * 1024 creation/deletion for files, 4GiB RAM, 3 execute runs).

I narrowed the regression to this window:

GENERIC 2011-07-03
seq_create ran_create ran_stat (#/s)
4374 4499 6052
4488 4553 6008
4616 4574 6057
GENERIC 2011-07-05
seq_create ran_create ran_stat (#/s)
1518 1539 1698
1534 1521 1689
1532 1541 1713

Somewhere between 2011-07-03 and -05, a commit resulted in about a cut by 3 in 
terms of files creation/deletion for a ffs file-system. I don't know yet 
exactly which commit is at fault, but I think that a 3 days window should be 
fairly fast to analyze.
>How-To-Repeat:
Compile a GENERIC kernel between 2011-07-03 and 2011-07-05, compile and install 
benchmarks/bonnie++, and run it:

bonnie++ -n 32 -r 4096 -x 3 -u nobody

Look for the fields that represent file sequential creation, randomized 
creation, and stat(2) (fields 16, 22 and 24):

=> awk -F, '{print $16 " " $22 " " $24}'
>Fix:
Unknown. Likely a locking change in the vfs/ffs layer that affects performance 
of ffs file-system in the 2011-07-03 <> 07-05 time frame.