Re: Why NetBSD x86's bus_space_barrier does not use [sml]fence?

[Taylor R Campbell <campbell+netbsd-tech-kern%mumble.net@localhost>]

> Date: Mon, 2 Dec 2019 15:54:34 +0900
> From: Masanobu SAITOH <msaitoh%execsw.org@localhost>
> 
> On 2019/12/02 15:48, s ymgch wrote:
> >> I had a misunderstanding. The driver uses BUS_SPACE_MAP_PREFETCHABLE
> >> specified in PCI configuration space.
> 
> This is well known pitfall:
> 
> 	http://mail-index.netbsd.org/tech-kern/2017/03/22/msg021678.html
> 
> I think we should change pci_mapreg_map().

I agree.  I committed the change you discussed to pci_mapreg_map
(option A, which seemed most sensible to me and seemed to have the
most support).

I also changed x86 bus_space_barrier to issue LFENCE/SFENCE/MFENCE,
which the AMD documentation clearly indicates (and the Intel
documentation unclearly indicates) is necessary for WC-type memory
regions, which are how we implement BUS_SPACE_MAP_PREFETCHABLE.

If these broke anything, please feel free to back out and reassess,
but I would like to have the semantics clearly documented.


P.S.  While studying this, it seemed to me that the fences in x86
bus_dmamap_sync should _not_ be necessary, because those are (as far
as I know) _never_ WC-type memory, and store-before-load is not a
relevant ordering for bus_dmamap_sync.  But, there have been some
vague references to symptoms in practice that were resolved by adding
the fences:

https://mail-index.NetBSD.org/port-amd64/2017/10/27/msg002575.html
https://gnats.NetBSD.org/21665
https://gnats.NetBSD.org/38935

So maybe I misunderstand something about x86 bus_dmamap_sync, or maybe
the hardware does not match the documentation.