[ Man, I want this thread to die, but I also feel as if I need to speak
  up, since this mail says things like "visualization" and "Cray" ...
  We have 3 Crays in the machine room used to generate visualization
  sets here :-) ]

On Wed, 23 Oct 1996 23:50:47 -0400 
 "Perry E. Metzger" <perry@piermont.com> wrote:

 > > Tell that to the millions of ISP's out there, data mining centers
 > > where an NFS server must be able to keep up with a CRAY over 2
 > > kilometers of fiber.
 > 
 > If you are using NFS in a high performance application you're a
 > fool. (I've told this to several ISPs, by the way.) Algorithm problems
 > again, you know. (BTW, the number of kilometers of fiber doesn't
 > appreciably change the problem.)  Switching to AFS, or a dedicated
 > protocol to feed data in and out of your cray, would make vastly more
 > sense than using NFS. Most Crays get fed by amazing disk and tape
 > arrays, by the way -- not NFS.

I'm with Perry ... if you think using NFS is good for data mining,
and keeping up with a Cray, well, I'm certainly glad you're not doing
system integration in my environment.

Our Crays (2 C90s; we have a J90 cluster, too, but I haven't really
played with it much, yet :-) load/store data from/to 2 _really huge_
`RAM disks' (well, that's basically what they are), which are eventually
flushed out to Max Strat disk arrays attached via Hippi-800.  NFS does
not enter the picture here; the 8k transactions would take away any
advantage gained by the Hippi (which imposes no MTU).

As far as the "2 kilometers of fiber" goes, I'm assuming you're referring
to serial Hippi... Unless someone has an ATM interface for Crays now,
that works properly.  Though, ATM isn't going to provide the bandwidth
necessary for these applications...

[ dave sez ]
 > > People doing next generation visualization and simulation over NFS
 > > using reality engines and getting real time,

Uhh ... Our facility is all about visualization ... if you suggested
using NFS to any of the vis group here, you'd be laughed at.  We need
to sustain over 300 MB/sec from the filesystem for our Virtual Windtunnel.
I forget the timestep granularity; I'm not a graphics weenie, but I can
ask someone who is... I do know, however, that the amount of data in one
timestep is quite large.

NFS just isn't going to provide that, period.  To even have that much
network bandwidth, you'd have to have at least 3 Hippi interfaces
running full-tilt on the system (which means that your NFS server would
also have to have at least 3 Hippi interfaces, and you'd have to either
put no one else on that network, or have at least 3 Hippi switches).
But, even then, there'd be the RPC reply, which would require one to
re-arbitrate the switch, send the reply (a very short message; bad for
Hippi), then re-arbitrate for the next transaction.  Repeat N times.

In other words, NFS would suck.  (We all know what NFS _really_ stands
for, right?)

The system that runs the VWT is a _big_ Power Onyx, with 4 (I think)
processors (I forget exactly which MIPS variant is in it), 2 Reality
Engines (one for each eye in the boom :-), a few gig of RAM, and
24 SCSI busses.

In any case, it's clear that the solution in the visualization case
is to use a protocol other than NFS (best to use local disk... doing
that over a distance is going to be much easier with Fibre Channel;
Hippi-6400 has a 1KM limit, I think...)

Put another way, you IMPROVE THE ALGORITHMS.

Jason R. Thorpe                                       thorpej@nas.nasa.gov
NASA Ames Research Center                               Home: 408.866.1912
NAS: M/S 258-6                                          Work: 415.604.0935
Moffett Field, CA 94035                                Pager: 415.428.6939