Re: Fixing excessive shootdowns during FFS read (kern/53124) on x86

[Maxime Villard <max%m00nbsd.net@localhost>]

Le 25/03/2018 à 17:03, Michael van Elst a écrit :
> On Sun, Mar 25, 2018 at 03:39:21PM +0200, Maxime Villard wrote:
> > The main cpu (on which your program executes) sends IPIs to remote cpus, these
> > cpus were idle and entered the halted state, and they take some time to wake
> > up and process the IPI. And that's time the main cpu spends waiting.
> >
> > As you said yourself it's not trivial to fix this, because the wakeup path
> > can be whatever interrupt entry point, and if we were to add some code there
> > it would slow down interrupt entry in general.
>
> The slow down would be minimal, the cost comes from refilling the TLB and
> that would happen independent on when the TLB was flushed.

I wouldn't be so sure; a new branch in the interrupt entry points (with
interrupts disabled) does cost something, and it's about the hottest path in
the kernel.

> > We could perhaps have something to instruct the main cpu not to wait for cpus
> > that are idle, only wait for those that aren't.
>
> Is there a guarantee that the IPI handler on the other CPUs has already
> started?

I don't think so, but I would have to re-check the LAPIC spec.

> Could it be deferred by some other high-level interrupt or
> by a section with disabled interrupts?

I don't think so (and I'm also not totally sure what you mean).

In all cases you need to handle the wakeup in the interrupt entry points.

An efficient way would be, before going into hlt, to flush the tlb and to
set a "idle" flag somewhere in curcpu. Then, we change each interrupt entry
point to set this flag to zero automatically regardless of its previous
value. Finally, when a cpu performs a tlb shootdown, it does not send IPIs
to the cpus flagged as idle.

Maxime