Very slow file write/create operations

[Frank Wille <frank%phoenix.owl.de@localhost>]

Hi,

I have 11.0RC2 installed on a HP ProLiant DL360p G8, with 128GB RAM and
RAID-1 via ciss(4).

# dmesg | grep ciss0
[     1.020058] ciss0 at pci5 dev 0 function 0: HP Smart Array 12
[     1.020058] ciss0: interrupting at msix4 vec 0
[     1.020058] ciss0: 1 LD, HW rev 1, FW 8.00/8.00, 64bit fifo rro, 
method perf 0x20000005
[     1.020058] scsibus0 at ciss0: 1 target, 1 lun per target

# bioctl ciss0 show
Volume Status       Size         Device/Label    Level Stripe
=============================================================
     0 Online       466G                 CISS   RAID 1    N/A
   0:0 Online       466G         0:0.0 noencl <ATA     TOSHIBA DT01ACA0>
   0:1 Online       466G         0:1.0 noencl <ATA     TOSHIBA DT01ACA0>


Already during installation I noticed that unpacking the archives takes
extremely long (hours). So I made a simple performance test with "cp"
and "rm" on the /etc directory:

# time cp -Rp /etc /root/
       20.20 real         0.00 user         0.06 sys
# time rm -rf /root/etc
       13.17 real         0.00 user         0.04 sys

Which is strange in two ways:
1. This is really slow! My older server (DL360 G5) did both in
Milliseconds!
2. Shouldn't the user- and sys-times more or less add into "real"? Where
did the system waste all that time?

On the other hand, creating a single 100MB file is quick as expected:
# dd if=/dev/zero of=/root/dummy bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 0.116 secs (903944827 bytes/sec)

Also reading is fast and writing to tmpfs:
# time cp -Rp /etc /tmp/
        0.03 real         0.00 user         0.03 sys
# time rm -rf /tmp/etc
        0.02 real         0.00 user         0.02 sys

I also compiled benchmarks/blogbench from pkgsrc, which should simulate
a busy file server. It gave the following result:
# blogbench -d /root/testdir
...
Final score for writes:            16
Final score for reads :        227925

Writes are more than factor 100000 slower!?

For comparison my old server had:
Final score for writes:           600
Final score for reads :         68988

Slower writes as well, but only factor 100.

What is wrong here? Does anybody have an idea what to check? Is it the
RAID or the ciss(4) driver? I remember I had similar results with 10.1.
The old server is running 8.3.

--
Frank Wille