Re: RFC: IRQ affinity (aka interrupt routing)

[Kengo NAKAHARA <k-nakahara%iij.ad.jp@localhost>]

Hi dyoung,

(2014/07/26 1:41), David Young wrote:
> On Fri, Jul 25, 2014 at 07:15:02PM +0900, Kengo NAKAHARA wrote:
> > Hi martin,
> >
> > Thank you very much for your comment.
> >
> > (2014/07/25 18:15), Martin Husemann wrote:
> > > A few general comments:
> > >
> > >   - best UI is no UI - the kernel should distribute interrupts automatically
> > >     (if possible) as fair as possible over time doiing some statistics
> >
> > I agree the computer should distribute interrupts automatically, but
> > I think balancing interrupts is too complex for the kernel. So I think
> > the balancing should be done by the userland daemon which use the UI.
> > Implementing and tuning the userland daemon are future works.
>
> What's the goal of balancing interrupts?  Controlling latency?  That's
> important, but it seems like other considerations might apply.  For
> example, funneling all interrupts to one core might allow the other
> cores to idle in a power-saving state.  Also, it might help to avoid
> cacheline motion for two interrupts involved in the same network flow to
> fire on the same CPU.

To be honest, I don't decide exactly the final goal yet, but I decide
the first milestone is improving the interrupt response in SMP systems
to reduce dropped packets.
# additionally, one of my motivation is helping tests of MPSAFE L2
# networking. See 
http://mail-index.netbsd.org/tech-kern/2014/06/03/msg017190.html
As you point out, there are the cases which all interrupts should be
funneled to one core. Whereas there are the cases which interrups
should be distributed, such as heavy workload servers and routers.
So, the system shoud have options for the administrators to select
how to balance interrupts.

> Can you explain why you think that balancing interrupts is too complex
> for the kernel?  I would not necessarily disagree, but it seems like
> the kernel has the most immediate access to the relevant interrupt
> statistics *and* the responsibility for CPU scheduling, so it's in a
> pretty good position to react to imbalances.

I think analyzing interrupt rates, deciding which IRQ's interrupts
shuld be moved and deciding which CPU is assigned the moved interrupts
are complex. As you point out, the kernel is good position to take
statistics. However, the tasks after taking statistics would take
long time. Those tasks are calculating which interrupts increase or
decrease in the previous seconds(, milliseconds, or minutes) and
deciding which cpu is the most "appropriate".
# The cores in a power-saving state may be not appropriate, and two
# interrupts involved in the same network are appropriate to route
# the same CPU, you know :)


Thanks,

--
//////////////////////////////////////////////////////////////////////
Internet Initiative Japan Inc.

Device Engineering Section,
Core Product Development Department,
Product Division,
Technology Unit

Kengo NAKAHARA <k-nakahara%iij.ad.jp@localhost>