[src/trunk]: src/sys/dev/pci Use BUS_SPACE_MAP_PREFETCHABLE only if BAR and d...

[riastradh <riastradh%NetBSD.org@localhost>]

details:   https://anonhg.NetBSD.org/src/rev/34e1aece0510
branches:  trunk
changeset: 846818:34e1aece0510
user:      riastradh <riastradh%NetBSD.org@localhost>
date:      Mon Dec 02 08:33:42 2019 +0000

description:
Use BUS_SPACE_MAP_PREFETCHABLE only if BAR and driver agree on it.

- A driver that expects prefetchable memory and knows to issue the
  needed bus_space_barrier calls can pass BUS_SPACE_MAP_PREFETCHABLE
  to indicate a desire to map the memory prefetchable if the BAR
  allows it.

  (A driver that _really wants_ BUS_SPACE_MAP_PREFETCHABLE even if
  the BAR claims _not_ to be prefetchable can use pci_mapreg_info and
  bus_space_map explicitly -- this is not different from what we have
  today.)

- For a driver that _does not_ expect prefetchable memory, the
  appearance of the prefetchable bit in the BAR shouldn't cause it to
  use BUS_SPACE_MAP_PREFETCHABLE, because the driver will not issue
  the needed bus_space_barrier calls to get sensible results.

Note: `Prefetchable' here, sometimes called `write-combining', means
reads have no side effects, and writes are idempotent, so it is safe
to issue reads out of order and safe to combine writes.

Mappings with BUS_SPACE_MAP_PREFETCHABLE are often more weakly
ordered than normal memory -- e.g., on x86, in WC-type memory
regions, loads can be reordered with loads, stores can be reordered
with stores, which is not possible with any other type of memory
regions.

Discussed on tech-kern a while ago:

https://mail-index.NetBSD.org/tech-kern/2017/03/22/msg021678.html

This is option A, which received the most support.  This should help
unconfuse drivers that do not expect prefetchable mappings, like
Yamaguchi-san tripped over recently:

https://mail-index.NetBSD.org/tech-kern/2019/12/02/msg025785.html

diffstat:

 sys/dev/pci/pci_map.c |  24 +++++++++++++++++++++---
 1 files changed, 21 insertions(+), 3 deletions(-)

diffs (45 lines):

diff -r b22b50d6d291 -r 34e1aece0510 sys/dev/pci/pci_map.c
--- a/sys/dev/pci/pci_map.c     Mon Dec 02 03:06:51 2019 +0000
+++ b/sys/dev/pci/pci_map.c     Mon Dec 02 08:33:42 2019 +0000
@@ -1,4 +1,4 @@
-/*     $NetBSD: pci_map.c,v 1.37 2019/03/01 05:41:56 msaitoh Exp $     */
+/*     $NetBSD: pci_map.c,v 1.38 2019/12/02 08:33:42 riastradh Exp $   */
 
 /*-
  * Copyright (c) 1998, 2000 The NetBSD Foundation, Inc.
@@ -34,7 +34,7 @@
  */
 
 #include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: pci_map.c,v 1.37 2019/03/01 05:41:56 msaitoh Exp $");
+__KERNEL_RCSID(0, "$NetBSD: pci_map.c,v 1.38 2019/12/02 08:33:42 riastradh Exp $");
 
 #include <sys/param.h>
 #include <sys/systm.h>
@@ -328,7 +328,25 @@
        if (realmaxsize < (offset + reqsize))
                return 1;
 
-       if (bus_space_map(tag, base, reqsize, busflags | flags, &handle))
+       /*
+        * If the BAR doesn't say prefetchable, assume the mapping
+        * cannot be prefetchable.  In other words, we will use
+        * BUS_SPACE_MAP_PREFETCHABLE only if the driver requested it
+        * _and_ the BAR indicates it is prefetchable.
+        *
+        * Drivers that expect prefetchable memory will issue the
+        * appropriate barriers, whereas drivers that do not expect
+        * prefetchable memory will be terribly confused if the memory
+        * is mapped prefetchable.
+        *
+        * Drivers that want to override the BAR and map it with
+        * BUS_SPACE_MAP_PREFETCHABLE anyway can use pci_mapreg_map and
+        * bus_space_map explicitly.
+        */
+       if (!ISSET(flags, BUS_SPACE_MAP_PREFETCHABLE))
+               busflags &= ~BUS_SPACE_MAP_PREFETCHABLE;
+
+       if (bus_space_map(tag, base, reqsize, busflags, &handle))
                return 1;
 
        if (pci_mapreg_map_enable_decode) {