Re: kern/60144: virtio(4) cache coherence issue

[Robert Elz <kre%munnari.OZ.AU@localhost>]

    Date:        Sun, 29 Mar 2026 21:39:43 -0700
    From:        Jason Thorpe <thorpej%me.com@localhost>
    Message-ID:  <52E223DA-EF6B-4266-836F-29955CE7FAAC%me.com@localhost>

  | This actually brings up an interesting philosophical question:
  | What does DMA coherency mean in the context of virtio?

Well, as you suggest, but didn't explicitly say, caching in a virtual
device can't logically be to make anything run faster, so the obvious
interpretation would be that it is intended to assist in finding caching
bugs.

If I were implementing something like that (and I certainly am not), I'd
be modelling all the memory the process generally sees (RAM) as cache,
ie: 100% cache coverage -- all ram is always in "cache".

Then I'd model real ram as only ever used by operations that explicitly
don't use the cache - like device DMA, but possibly also interrupt vector
fetch - anything which the modelled hardware doesn't pass by the cache
(and for devices with multiple cache levels, that means all of them.)

For that I'd only allocate real memory for the non-cache pages which are
actually used - when one is first allocated, I'd simply copy the "cached"
memory into it (as if at that instant, for whatever reason, the cache was
completely consistent) - but after that the only operations which would
ever move data between the external memory (where it has been allocated) and
the cache memory would be those which explicitly affect the cache in the
hardware design - operations which flush a cache line to ram (in real hardware)
would copy that amount of data from the cache to the external ram, and
operations which invalidate the cache (without writeback) would simply
copy the external ram into the cache memory (as if the cache had been
invalidated, and then something magically referenced the relevant cache
lines, and copied the ram data into the cache).   But both only when there
is an actual ram page allocated, when not, the cache ops would just be
no-ops.

When modelling a multi-cpu system with caching, I'd allocate entirely
separate memory for each cpu (its cache), but share memory for the external
ram - so any ram intended to be visible to multiple CPUs needs the cache
flush logic implemented properly - in this case I'd probably allocate the
ram backing pages for anything which might be shared between cpus, as well
as when used for dma etc.

So, I'm not sure that the general concept is quite as absurd as it seems
at first glance - it might help locate all kinds of caching issues such
as the one Taylor described.

kre

ps: everyone, when replying to gnats messages, it is a good idea, I
believe, to delete the gnats-admin@ address from the destination fields.
Having it there serves no useful purpose as best I understand things.