Hi All,

I recently received the thread below, after some serious searching for
solutions to our DMA problems - if you use the above combination of chips in
any systems, please let me know if the thread below looks like a serious
threat!

Thanks
 Bruce

-
>-----------------------------
>I'd like to address the issue of data caching on the ebsa285.
>(kernel is 2.2.10-rmk2)
>
>We have been running our ebsa285s with data caching off. With caching
>disabled
>the stock ncr53c8xx SCSI driver works well and the lance ethernet driver
>works
>nicely w/ a few changes. Everything falls apart, however, when we turn
>on
>data caching. The NCR scsi shuts its self down after detecting
>cache coherency problems, and the lance ethernet simply does not work.
>
>In a naive attempt to solve the problem we've tried several combinations
>of
>the following:
> 1.) There is a driver out there called bigphysmem; it allocates a fixed
>
> chunks of memory at boot time and provides a malloc() interface to it.
> We turn off caching on the sections managed by this driver.
> We modified the scsi and ether drivers to use the bigphysmem malloc().
>
> 2.) we modified kmalloc et.al. to turn off caching on the
> returned pages.
>
>With any combination of these changes the kernel will boot
>and the drives work reasonably well, but not 100%.
>
>One class of problems seems to be that some file are corrupted.
>for example the shell complains when it tries to source in
>its config files:
>
>bash: .: /etc/profile.d/lang.sh: cannot execute binary file
>
>With caching turned off, the bigphysmem modified drivers
>work as expected (it boots w/ no errors).
>
>
>Finally to the questions:
> 1.) Is anybody actually running an ebsa285
>  with caching enabled ?
>
>  If so what type of changes are
>  required to the drivers ?
>
> 2.) Are we way off base here ?
>  Shouldn't the bigphysmem or changes to kmalloc()
>  solve the problem ?
>
>  What subtlety are we missing ?
>
>
>
>--
>dave.baukus@inet.com
> Inet Technologies Inc.
> Plano, Texas
>--------------------------------------------------------------------------
>
>Subject: Re: EBSA285 and Cacheing
>In-Reply-To: <37D8370C.A5499569@inet.com>
>References: <199812191722.RAA01668@raistlin.armlinux.org>
> <37D8370C.A5499569@inet.com>
>Sender: owner-linux-arm@vger.rutgers.edu
>Precedence: bulk
>X-Loop: majordomo@vger.rutgers.edu
>X-Orcpt: rfc822;linux-arm-outgoing
>
>We use an EBSA-285 with caching on.  You have to be careful which
>drivers you use and any references to DMA data must be protected by
>cache flushes.  Look at the tulip.c driver -  you'll see an example of
>all the cache flushes used.  You'll need to modify any of your drivers
>which use DMA to behave in a similar way.  Basically, you need to flush
>the cache immediately after setting up data structures read by the
>device and you need to invalidate the cache before reading those data
>structures.
>
>-Jon
>
>Excerpts from linux-arm: 9-Sep-99 EBSA285 and Cacheing by Dave
>Baukus@inet.com
>> I'd like to address the issue of data caching on the ebsa285.
>> (kernel is 2.2.10-rmk2)
>>
>> We have been running our ebsa285s with data caching off. With caching
>disabled
>> the stock ncr53c8xx SCSI driver works well and the lance ethernet
>driver works
>> nicely w/ a few changes. Everything falls apart, however, when we turn
>on
>> data caching. The NCR scsi shuts its self down after detecting
>> cache coherency problems, and the lance ethernet simply does not work.
>
>>
>> In a naive attempt to solve the problem we've tried several
>combinations of
>> the following:
>>         1.) There is a driver out there called bigphysmem; it
>allocates a fixed
>>         chunks of memory at boot time and provides a malloc()
>interface to it.
>>         We turn off caching on the sections managed by this driver.
>>         We modified the scsi and ether drivers to use the bigphysmem
>malloc().
>>
>>         2.) we modified kmalloc et.al. to turn off caching on the
>>         returned pages.
>>
>> With any combination of these changes the kernel will boot
>> and the drives work reasonably well, but not 100%.
>>
>> One class of problems seems to be that some file are corrupted.
>> for example the shell complains when it tries to source in
>> its config files:
>>
>> bash: .: /etc/profile.d/lang.sh: cannot execute binary file
>>
>> With caching turned off, the bigphysmem modified drivers
>> work as expected (it boots w/ no errors).
>>
>>
>> Finally to the questions:
>>         1.)     Is anybody actually running an ebsa285
>>                 with caching enabled ?
>>
>>                 If so what type of changes are
>>                 required to the drivers ?
>>
>>         2.)     Are we way off base here ?
>>                 Shouldn't the bigphysmem or changes to kmalloc()
>>                 solve the problem ?
>>
>>                 What subtlety are we missing ?
>>
>>
>>
>---------------------------------------------------------------------------
----
>
>>Finally to the questions:
>> 1.) Is anybody actually running an ebsa285
>>  with caching enabled ?
>>
>>  If so what type of changes are
>>  required to the drivers ?
>>
>> 2.) Are we way off base here ?
>>  Shouldn't the bigphysmem or changes to kmalloc()
>>  solve the problem ?
>>
>>  What subtlety are we missing ?
>
>This is a generic problem with all SA110/21285 platform (EBSA,
>Netwinder,
>CATS, ...)
>
>I discussed this recently with Philip and Russel. The problem is that
>the
>SA110/21285 pair doesn't handle cache coherency. That means that when a
>devices accesses the memory via the PCI (bus master), nothing is done to
>
>make sure the datas are coherent with the CPU cache.
>(The architecture of the SA110 cache, based on virtual addresses, would
>probably have made this almost impossible without a bunch of hacks in
>the
>silicon).
>
>Some drivers have been "fixed" for this behaviour by adding various
>cache
>flushes and invalidate on memory ranges at specific locations.
>Basically,
>you need to invalidate a range which will be written by the device and
>flush a range which will be read by the device. This is how the tulip
>driver was fixed and I did fix the pcnet32 driver the same way recently
>(I can send you patches). I've seen a fixed sym53c8xx in the CVS
>recently.
>
>The problem with this solution is that it's far from perfect. It imposes
>
>to place those flush and invalidate in critical locations which can make
>
>the drivers quite unmaintainable. Also, when dealing with "shared
>regions" where both the CPU and the device will read&write small datas
>(words), there are still some potential coherency issues. The CPU
>flushes
>cache lines, so when flushing, let's say, a word, the entire cache line
>containing this word will be flushed, possibly corrupting whatever the
>device wanted to write somewhere else in this same cache line. The
>current fixed tulip and pcnet32 seems to work fine, probably because the
>
>SA110 support a mecanism of "half cache lines" (2 dirty bits per line),
>making the half cache line the same size as a ring descriptor entry for
>those drivers.
>
>However, a better solution (and probably more efficient, especially for
>the symbios driver) would be to allocate those shared regions in
>non-cachable space. Flushes and invalidate would still be required for
>data buffers, but ring descriptors or SCSI controller scripts should be
>in non-cachable space.
>
>I tried implementing a vmalloc_uncached (with the help of Russel) but
>unfortunately, virt_to_bus can't get the physical address of a
>vmalloc'ed
>area, and I didn't want to walk the page tables.
>
>I had to stop hacking on this (more urgent work to do), but I beleive
>there is still the possibility to use the low-level _ioremap function to
>
>create a second, uncachable, mapping for a given kmalloc'ed region. If
>you make sure to invalidate it once, and then only use the address
>returned by _ioremap, this should work.
>
>I'll do more experiments with this next week,
>
>Benjamin.
>
>---------------------------------------------------------------------------
----------
>
>Subject: Re: EBSA285 and Cacheing
>Sender: owner-linux-arm@vger.rutgers.edu
>Precedence: bulk
>X-Loop: majordomo@vger.rutgers.edu
>X-Orcpt: rfc822;linux-arm-outgoing
>
> I tried implementing a vmalloc_uncached (with the help of
> Russel) but unfortunately, virt_to_bus can't get the physical
> address of a vmalloc'ed area, and I didn't want to walk the page
> tables.
>
> I had to stop hacking on this (more urgent work to do), but I
> beleive there is still the possibility to use the low-level
> _ioremap function to create a second, uncachable, mapping for a
> given kmalloc'ed region. If you make sure to invalidate it once,
> and then only use the address returned by _ioremap, this should
> work.
>
> I'll do more experiments with this next week,
>
>A student here at the UT told me he had this working last week. I didn't
>
>have the time yet to test it but I hope I'll be able to test it this
>week.
>
>greetings,
>
>Mark.
>---------------------------------------------------------------------------
----------
>
>Subject: Re: EBSA285 and Cacheing
>In-Reply-To: <199909122154.XAA30588@poekie.test>
>Message-ID: <Pine.GSO.4.10.9909131023460.8943-100000@isis.visi.com>
>MIME-Version: 1.0
>Content-Type: TEXT/PLAIN; charset=US-ASCII
>Sender: owner-linux-arm@vger.rutgers.edu
>Precedence: bulk
>X-Loop: majordomo@vger.rutgers.edu
>X-Orcpt: rfc822;linux-arm-outgoing
>
>Hi,
>
>On Sun, 12 Sep 1999, Mark van Doesburg wrote:
>
>>  I tried implementing a vmalloc_uncached (with the help of
>>  Russel) but unfortunately, virt_to_bus can't get the physical
>>  address of a vmalloc'ed area, and I didn't want to walk the page
>>  tables.
>>
>>  I had to stop hacking on this (more urgent work to do), but I
>>  beleive there is still the possibility to use the low-level
>>  _ioremap function to create a second, uncachable, mapping for a
>>  given kmalloc'ed region. If you make sure to invalidate it once,
>>  and then only use the address returned by _ioremap, this should
>>  work.
>>
>>  I'll do more experiments with this next week,
>>
>> A student here at the UT told me he had this working last week. I
>didn't
>> have the time yet to test it but I hope I'll be able to test it this
>> week.
>>
>For uncacheable memory, I talked with Phil Blundell about this, and he
>pointed me at the framebuffer for the sa1100 linuxarm port.  I pulled
>the bits out of there that grabbed memory in the kernel and marked it
>uncacheable, and made it a separate routine.  I haven't tested it yet
>other than to verify the call suceeds and memory is returned.  I can
>post the routine to the list if anyone is interested.
>-----------------------------------------------------------------
>