Subject: Alternative to memory maps for large (> 10GB) files on ix86.
To: None <port-i386@netbsd.org>
From: Alicia da Conceicao <alicia@cyberstation.ca>
List: port-i386
Date: 10/16/2000 01:18:45
Memory maps provide the most efficent means of randomly accessing data
within a file.  However they cannot be used for a file that exceed the
available address space for a process (2 GB - 1B), used by signed 32 bit
pointers.  In fact the available address space has to be shared by all
memory mapped files loaded by the process.

Is there an efficent alternative to memory maps for huge (> 10 GB) files
on NetBSD ix86, that can be used for many random database accesses.  I am
hoping for a solution which doesn't involve many, many seeks & reads, or
constant loading & unloading of memory maps containing small sections of
each file?

Yes I know that 64 bit hardware like Sparc64, Alpha, & Mac can memory map
huge files, but I need a solution for 32 bit ix86 hardware, since these
are to be deployed on the client side with cheap, existing PC computers.

Writing alternatives to tail & more/less that work using seek & read
instead of memory maps would be quite easy to do.  But for applications
that needs to read large amounts of data in random order from multiple 
huge (> 10 GB) files, using many, many seeks & reads would be too
inefficent.
 
Any advice on an efficent NetBSD ix86 memory map alternative for huge
files would be appreciated.  Maybe combining some type of elaborate
caching with seek & reads, like that which is done at the kernel level 
with mmap's "SYS_mmap" system call?
 
Thanks in advance. 
Alicia.