PCI BAR's prefetchable bit and pci_mapreg_map()

[Masanobu SAITOH <msaitoh%execsw.org@localhost>]

 Hi.

 As some people know, our pci(9) had a problem with devices that
a BAR has prefetchable bit. It makes unexpected behavior on read/
write. To avoid this problem some device drivers have common hack.
Instead of using pci_mapreg_map(), it uses pci_mapreg_info() and
pass the result to bus_space_map() without BUS_SPACE_MAPPREFETCHABLE.
You can see it in:
	pci/if_fxp_pci.c
	pci/if_bge.c
	pci/ixgbe.c
	pci/ixv.c
	x86/pci/msipic.c

Example in if_bge.c:
	http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/dev/pci/if_bge.c.diff?r1=1.272&r2=1.275&f=h

 I found the cause in PCIe's specification. It's required to pay
money to get the specification, but it's not required for ECN and
Errata. You can see the information about prefetchable bit in
"Errata for the PCI Express(R) Base Specification Revision 2.1"
(November 18, 2010):

	https://pcisig.com/specifications/pciexpress/specifications?field_technology_value%5B%5D=express&field_document_type_value%5B%5D=errata&speclib=

See

	1.3.2.2 PCI Express Endpoint Rules

	7.5.2.1 Base Address Register (offset 20h - 24h)

	IMPLEMENTATION NOTE in 7.5.2.1 "Additional Guidance on the Prefetchable
	Bit in Memory Space BARs" (<==== very important)

It says that the prefetchable bit can be set even if the range
include read side effect or effect of write combine...........

 Our pci_map.c::pci_mapreg_map() first does pci_mem_find() and
the result is used for does bus_space_map() with BUS_SPACE_MAP_PREFETCHABLE
if a BAR has prefetchable bit. For x86, it result in adding
PMAP_WRITE_COMBINE in the map. cardbus_mem_find() does the same thing.

On other OSes:

 Linux:
	It has ioremap() and ioremap_wc(). ioremap_wc() is used
	in a driver only if the driver know the whole area
	specified in a BAR is really prefetchable. ioremap() doesn't
	set the area prefetchable.

 FreeBSD:
	 After mapping a area with bus_alloc_resources_any(), if
	the driver know the whole area is really prefetchable,
	pmap_change_attr() is called with PAT_WRITE_COMBINE.

 OpenBSD:
	 It ignores prefetchable bit in pci_mapreg_map()
		http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/dev/pci/pci_map.c.diff?r1=1.28&r2=1.29&f=h


 So what should we do?

 A) modify pci_mapreg_map(). Stop setting BUS_SPACE_MAP_PREFETCHABLE
    by defalut when prefetchable bit is set. If a driver really know
    the whole area of the BAR is prefetchable, set BUS_SPACE_MAP_PREFETCHABLE
    in the 4th argument(busflags) of pci_mapreg_map(). pci_mapreg_map()
    check for both the 4th argument and the prefetchable bit, it sets
    BUS_SPACE_MAP_PREFETCHABLE only when both bits are set.

 B) Don't modify pci_mapreg_ma(). Add new API like FreeBSD's pmap_change_attr()
    and use it when a driver want.

 C) make new function pci_mapreg_map_wc()

 D) Keep. If a driver know a BAR has prefetchable bit and not the
    whole area is prefetchable, do pci_mem_find() -> drop BUS_SPACE_MAP_PREFETCHABLE
    and do bus_space_map().

 E) Any other solution(s).

 Usually, framebuffer of video really uses prefetchable access.
Not so many driver use pci_mapreg_map() and use pci_mem_find() and
bus_space_map(), so it's not difficult to change them.

I prefer A.

--
-----------------------------------------------
                SAITOH Masanobu (msaitoh%execsw.org@localhost
                                 msaitoh%netbsd.org@localhost)