The Promise Ultra33 and Ultra66 controllers have 2 channels per chip.  
Each channel can connect to two disks - master and slave.  I have 
a system with wd0 as master on channel 0 and wd1 as master on channel 
1.  The startup printout is below. 

I have a simple program that reads from /dev/wd0d as fast as it can.  
If nothing else is going on, it gets 135 megabits.  Similarly, I 
can read from /dev/wd1d at 135 megabits.  In both cases, top says 
the system is over 60% idle. 

If I read both disks at the same time (same program running in separate 
windows), I only get 85-88 megabits on each drive.  Top says there 
is 20% of the CPU idle.  The numbers vary slightly depending on the 
size of the buffer I pass to read but they are always way below 130 
megabits. 

Why aren't both drives reading at full speed?  Where is the bottleneck?  
What is the limiting resource?  Is there some lock I don't know about?

I'm expecting the rotational transfer rate to be the limiting factor.  
I think the drives will read ahead into an on-disk cache so throughput 
shouldn't degrade (much) if the system takes time out to go work 
on the other drive.  [This may be wrong, or it may not read ahead 
as much as I'm expecting.]

I can also read from another drive connected to the non-Ultra builtin 
IDE chip.  Again, it runs over 130 megabits when nothing else is 
happening but when I also read one of the Ultra drives I get 95+99 
megabits.  Humm.  There may be an important clue here.  The non Ultra 
case only gets 132 to 133 megabits while the faster controller gets 
135 or 136.  Maybe the read ahead isn't working as well as I expect. 


  pciide0 at pci0 dev 15 function 0: Promise Ultra66/ATA Bus Master IDE Accelerator
  pciide0: bus-master DMA support present
  pciide0: primary channel configured to native-PCI mode
  pciide0: using irq 11 for native-PCI interrupt
  wd0 at pciide0 channel 0 drive 0: <IBM-DJNA-372200>
  wd0: drive supports 16-sector pio transfers, lba addressing
  wd0: 21557MB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 44150400 sectors
  wd0: 32-bits data port
  wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 4
  wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 4 (using DMA data transfers)
  pciide0: secondary channel configured to native-PCI mode
  wd1 at pciide0 channel 1 drive 0: <IBM-DJNA-372200>
  wd1: drive supports 16-sector pio transfers, lba addressing
  wd1: 21557MB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 44150400 sectors
  wd1: 32-bits data port
  wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 4
  wd1(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 4 (using DMA data transfers)