[ thinking aloud for much of this, good bits at the end. ]

Wayne Knowles <w.knowles@niwa.cri.nz> writes:
> Thanks for the suggestions Chris.

no problem.


> After adding extra tests around the read(2) calls in dump that fill the
> last 4 bytes with a known value (0xbadface) and checking after the call
> prove that read is not filling the buffer in the last 4 bytes. 
> Plus, the returned byte count is the same as the requested count.

hmm.


> If the read is repeated then there is a 90% chance that is succeedes (I
> write a different value for the retry in case my magic number was in fact
> the correct value)

how "90%" is that 90%?  8-)

(i suppose 90% may well be correct, if it's TLB-related since the tlb
to evict when you need one is random.)


> In doing this test I observered a very useful peice of information:
> 
>   The buffer addresses that dump reads into are not aligned - in fact they
>   are offset by 4 bytes from an aligned address.  It also fails when I set
>   it to 8 (see below)

where does it fail when you set it to 8?  is it always the last 4
bytes, or in the 8-byte case is it the 4 previous to that?


> The result is an unaligned copyout - which fails if it is in the last 2k
> of the hardware pages (half of NBPG)

well, for some value of unaligned; there's nothing to say that it must
have any alignment at all.  8-)

and, in particular, for the bcopy code, the notion of "aligned" is
4 byte alignment.


> Thinking of a unaligned copyout bug I checked the code there and couldn't
> see any bugs.  It falls down to use bcopy (written in assembler) and it
> isolate that I replaced it with the kernel bcopy.c implemention.  No
> change in the fault.

that's interesting.  to me that means that there's not likely a bug in
the bcopy implementation, which is comforting.


i looked at the dump code a bit, it looks it's using a raw device, not
a block device.  if so, the BLKDEV_IOSIZE thing is a bit of a red
herring.

Raw devices cause user pages to be wired and the data to be DMA'd
directly.  One potentially interesting question here is, is the cache
flushing routing going to work right.  All of the mips cache flush
routines assume that the source and destination are line-aligned...


> In the above case if buf=0x20044804 or 08 or 0c and we do a 2k read it
> will fail.

how about greater values?  does one of your caches have 16-byte lines?

if the only values that work are < your cache line size, this is
probably a flushing bug.  (if not, it's something in fault or TLB
handling.)


what kinds of cpus are in the machines that you noticed this on?
(MIPS I, MIPS III, etc.)

some part of me is wondering if this is a cache problem, where the
flush routines are doing something wrong, but the r2k/r3k mips1 flush
routines look OK (line size seems irrelevant), and the mips3 code
seems to try to do the right thing w.r.t. making sure all lines really
are hit.


so, more information about your specific CPU and cache configuration
is necessary, i think, to find out exactly which of the cache rtns
you're using.

it seems to me that if it's not a cache problem, then it's gotta be
something weird happening when that next page is being mapped into
your address space, and for that i'd look at the 'pagefault' code in
trap.c...


cgd