Re: Revisiting uvm_loan() for 'direct' write pipes

[Jaromír Doleček <jaromir.dolecek%gmail.com@localhost>]

2018-05-25 23:19 GMT+02:00 Jason Thorpe <thorpej%me.com@localhost>:
> BTW, I was thinking about this, and I think you need to also handle the case
> where you try the ubc_uiomove_direct() case, but then *fall back* onto the
> non-direct case if some magic error is returned.
> ...
> These cache flushes could be potentially very expensive.

I think that it would not be actually performance-wise make sense to
do this - the decision to fallback would have to be done within the
pmap_direct_process() function, which happens only after all the
preparation steps have been already done. If we abort the operation
here and go call the non-direct code path, it could be even slower
than the cache flush. For UBC the fallback would not be too terrible
to code and debug, but fallback would be very messy e.g. for the
uvm_loan() in sys_pipe.c.

I don't know what to do for such archs, but I guess it would need to
be evaluated if the direct map could be used in a way which actually
improves performance there for the general case.

2018-05-26 16:52 GMT+02:00 Thor Lancelot Simon <tls%panix.com@localhost>:
> In this case, could we do better gathering the IPIs so the cost were
> amortized over many pages?

Yes, it could gather the IPI for this particular TLB flush. Seems e.g.
on my system it would be enough to gather at least 3-4 to match the
speed. There is some limit to how this could be scaled though - right
now we have only 16 TLB IPI slots, which would be enough for maybe 1
parallell pipe setup. I've noticed DragonflyBSD did some work on
asynchronous TLB IPIs, maybe it would be possible to look for
inspiration there.

I do plan some tweaks for the TLB flush IPIs. I want to change the x86
pmap to not trigger TLB flush IPIs to idle processors, as was
discussed earlier for kern/53124. This would reduce the performance
impact of the IPIs if there are many idle processors. Seems acpi
wakeup takes linear time depending on number of cpus - I think this is
also something we'd want to fix too.

That said, regardless which way we slice it, anything which triggers
IPIs or TLB flushes inherently just does not scale on MP systems. If
uvm_loan() is to be used as high-performance zero-copy optimization,
it simply must avoid those. Adding something which triggers interrupt
to all active processors for every processed page simply can't
perform.

Jaromir