Subject: Lousy filesystem performance, or is it my motherboard?
To: None <current-users@netbsd.org, port-i386@netbsd.org>
From: Thor Lancelot Simon <tls@cloud9.net>
List: port-i386
Date: 11/30/1994 13:05:33
After months of piddling around with awful IDE disks on my machine at home,
I went and plugged in the BT445S I'd bought with the machine, and bought
myself a Micropolis 1-gig disk.  (roughly equivalent to the ST312000N/
Fujitsu 2694/et al.; a 5400RPM, 8ms drive).

The first thing I noticed was that performance didn't seem much better than
it had been with the IDE disk, from a subjective point of view.  So, I went
and did some benchmarking with iozone.

On a type 2 UFS, reads maxed out around 500K/s, and writes around 600K/s,
once I was writing files big enough for the cacheing not to matter.  This
seemed Ungood.  One of my clients has a machine running FreeBSD 1.1R, with
an Adaptec 1742, and the same Micropolis drive I have, so I went over there
and ran iozone again.

Performance was _much_ better.  On his machine, I got about 2.5MB/s write, and
1MB/s read.  (figures are from memory -- actual iozone data appended below.)
I wasn't sure if this was a consequence of the operating system, the
controller, or perhaps that he has a reasonably high-quality EISA motherboard,
and I have an el-cheapo UMC VESA.  Both machines are 486/66 with 256k cache;
to make things even scarier, I did my tests on my machine in single-user
mode, but his is a busy news server, and I was running iozone on the disk the
news spool is on!

So I went home and realized I could likely localize the problem by running
iozone directly on the device.  I did that with both the raw and block
devices; the block device gave somewhat better performance when everything
was just going into and out of cache, but at larger file sizes exhibited a
strange behaviour in which performance was reduced by an order of magnitude
when the record size was 4096.  I assume this has something to do with the
page size and the buffer cache?  In general, of course, the raw device
performed better.

My raw device scores were respectable -- 4MB/s write and 3MB/s read.  To
me, this would seem to indicate that the problem is in the UFS code somewhere
and not with my motherboard/SCSI controller.  Unfortunately, I wasn't able
to wipe out a partition on the FreeBSD machine to get raw scores for that
os/motherboard/scsi controller.

All my test data is appended below.  If anyone has any idea what's going on
here, I'd love to know.

I'm running NetBSD1.0 (1.0A now, but the scores are no better :-( ), with
type 2 filesystems.  Incidentally, while futzing around with this I think
I found an spl problem in the LFS code, if anyone cares.  Oh -- and one other
thing -- I noticed that the disklabel thought my disk was 3600RPM (the install
script should really *ask*, no?  Most new disks are 5400RPM at least by now),
but newfs-ing the filesystem in question with -r 5400 didn't help any.

IOZone was built with -DUSE_FSYNC.
-- IOZone data  for NetBSD 1.0A, UFS

	IOZONE: Performance Test of Sequential File I/O  --  V2.01 (10/21/94)
		By Bill Norcott

	Operating System: POSIX 1003.1-1990

IOZONE: auto-test mode 

	MB      reclen  bytes/sec written   bytes/sec read      
	1       512     1978445             294543              
	1       1024    2833989             284166              
	1       2048    3495253             281119              
	1       4096    4032984             278136              
	1       8192    5518821             504123              
	2       512     779610              499321              
	2       1024    896218              502914              
	2       2048    525602              496955              
	2       4096    533626              494611              
	2       8192    554802              498135              
	4       512     586615              494611              
	4       1024    607870              494611              
	4       2048    623224              494611              
	4       4096    632625              497544              
	4       8192    647269              494611              
	8       512     531934              494903              
	8       1024    539460              494903              
	8       2048    542600              492867              
	8       4096    545068              493447              
	8       8192    550433              494319              
	16      512     536527              495634              
	16      1024    536699              494611              
	16      2048    539807              494611              
	16      4096    542951              494028              
	16      8192    541375              495634              
Completed series of tests

--IOZone data for FreeBSD 1.1R, UFS

	IOZONE: Performance Test of Sequential File I/O  --  V2.01 (10/21/94)
		By Bill Norcott

	Operating System: POSIX 1003.1-1990

IOZONE: auto-test mode 

	MB      reclen  bytes/sec written   bytes/sec read      
	1       512     1066348             635500              
	1       1024    1429876             3932159             
	1       2048    2029501             1367707             
	1       4096    2419790             599186              
	1       8192    3311292             1187067             
	2       512     1113532             905245              
	2       1024    1850428             817072              
	2       2048    2374134             1066348             
	2       4096    2419790             526481              
	2       8192    3400787             344737              
	4       512     960527              401368              
	4       1024    1981560             809190              
	4       2048    2542002             731564              
	4       4096    1784810             1089429             
	4       8192    3447373             733697              
	8       512     1471685             1064094             
	8       1024    1973790             910156              
	8       2048    2443283             797648              
	8       4096    2594414             747869              
	8       8192    2197888             944308              
	16      512     1221641             1117239             
	16      1024    2071261             972592              
	16      2048    2062772             1072026             
	16      4096    1874549             1035630             
	16      8192    1857256             1122221             
Completed series of tests

--IOZone data for NetBSD 1.0A, no filesystem.

	IOZONE: Performance Test of Sequential File I/O  --  V2.01 (10/21/94)
		By Bill Norcott

	Operating System: POSIX 1003.1-1990

IOZONE: auto-test mode raw device /dev/rsd0g

	MB      reclen  bytes/sec written   bytes/sec read      
	1       4096    1263344             1278751             
	1       8192    2184533             2139951             
	1       16384   3495253             3084047             
	1       32768   4993219             3276800             
	1       65536   6168094             3615779             
	2       4096    1255779             1327311             
	2       8192    2207528             2255002             
	2       16384   3554494             3328812             
	2       32768   4993219             3554494             
	2       65536   4993219             3744914             
	4       4096    1255779             1340033             
	4       8192    2184533             2279513             
	4       16384   3554494             3382503             
	4       32768   4415056             3524625             
	4       65536   4032984             3847985             
	8       4096    1253902             1316892             
	8       8192    2190237             2310911             
	8       16384   3524625             3466366             
	8       32768   3994575             3615779             
	8       65536   3938313             3956890             
	16      4096    1255779             1348650             
	16      8192    2184533             2314098             
	16      16384   3532045             3600260             
	16      32768   3910772             3663147             
	16      65536   3787181             3947580             
Completed series of tests


	IOZONE: Performance Test of Sequential File I/O  --  V2.01 (10/21/94)
		By Bill Norcott

	Operating System: POSIX 1003.1-1990

IOZONE: auto-test mode raw device /dev/sd0g

	MB      reclen  bytes/sec written   bytes/sec read      
	1       4096    223101              2330168             
	1       8192    7489828             2330168             
	1       16384   7489828             2383127             
	1       32768   8065969             2383127             
	1       65536   7489828             2383127             
	2       4096    223576              2496609             
	2       8192    4877097             2438548             
	2       16384   4766254             2438548             
	2       32768   4766254             2467237             
	2       65536   4877097             2467237             
	4       4096    222627              2542002             
	4       8192    3554494             2496609             
	4       16384   3495253             2526689             
	4       32768   3495253             2542002             
	4       65536   3495253             2542002             
	8       4096    222568              2573192             
	8       8192    3072750             2534322             
	8       16384   3095427             2549728             
	8       32768   3072750             2557502             
	8       65536   3095427             2565323             
	16      4096    222627              2589076             
	16      8192    2922859             2557502             
	16      16384   2912711             2573192             
	16      32768   2907663             2577145             
	16      65536   2917776             2577145             
Completed series of tests