Interesting USB observation

[John Klos <john%ziaspace.com@localhost>]

Hi,

While this may be known and not that unexpected, I thought I'd share a 
little observation.

I have a quad core amd64 system running i386 NetBSD 5 (I haven't figured 
out how to get the code to run properly under amd64). I'm running some CPU 
intensive applications which will take almost a week and is generating a 
couple of terabytes of intermediate data. I was running low on space on 
one of the SATA drives, so I connected a 1 TB USB drive.

Observation 1: when scp'ing files from another machine to one of the SATA 
connected drives at 35 MB/sec, I started another scp to the same drive; 
the total rate went up to 50 MB/sec (which I think is all the drive can 
do). While just one of the scps was running, I started an scp to the USB 
drive. The total rate for both scps dropped to 10 MB/sec!

Observation 2: even when reading a modest amount of data from the USB 
drive (about 3 MB/sec), the CPU intensive tasks get slowed down 
significantly. Even though at least two of them are getting their data 
from the SATA drives, they're anywhere from 50% to 10% idle while using 
USB for anything. When I stop talking to the USB drives, they then run at 
96% to 98% CPU each.

While transferring over USB, system overhead for all four CPUs is less 
than 5% per CPU, and interrupt overhead is less than 2% for one CPU and 0% 
for the other three. With no USB tranfers and two tasks running off of a 
SATA drive transferring about 5 MB/sec, the total number of interrupts is 
around 200 to 250 a second. When running tasks or transferring over USB, 
that jumps to about 2000 to 2300!

I always knew that USB kinda sucked, but I had no idea it is this bad! 
Does anyone know if other OSes have this much of a performance impact when 
using USB? Is this just due to really horrible interrupt handling and 
overhead?

ohci0 at pci0 dev 18 function 0: vendor 0x1002 product 0x4397 (rev. 0x00)
ohci0: interrupting at ioapic0 pin 16
ohci0: OHCI version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
...
ehci0 at pci0 dev 18 function 2: vendor 0x1002 product 0x4396 (rev. 0x00)
ehci0: interrupting at ioapic0 pin 17
ehci0: dropped intr workaround enabled
ehci0: BIOS has given up ownership
ehci0: EHCI version 1.0
ehci0: companion controllers, 3 ports each: ohci0 ohci1
...

Thanks,
John Klos