Hi, (2014/07/25 18:06), Kengo NAKAHARA wrote:
I implement IRQ affinity (aka interrupt routing) for amd64. Here is
the implementation.
https://github.com/knakahara/netbsd-src/compare/rfc/irq-affinity
But the UI is rough, so could you comment aboud the UI?
Above implementation works well, but I think the UI is not so good.
So, I am going to implement new command intrctl(8). The usage is consist
of below two sub command.
(A) "intrctl list"
This sub command show interrupt numbers per cpu (like
/kern/interrupts).
(B) "intrctl -i 18 -c 1"
This sub command let interrupts route other cpu. "18" is IRQ
number, "1" is cpuid (like "sysctl -w kern.cpu_affinity.irq=18:1").
I implement "intrctl" for amd64 and i386. Here is the implementation,
https://github.com/knakahara/netbsd-src/compare/rfc/irq-affinity2
and here is the patch
http://knakahara.github.io/patches/netbsd/irq-affinity-initctl.patch
"intrctl" has 4 subcommand,
- list
for each IRQ in the system, display interrupt counts per CPU.
- affinity -c "cpuid" -i "irq"
set affinity "irq" interrupt to "cpuid".
- intr -c "cpuid"
enable interrupts to set affinity to cpuid.
- nointr -c "cpuid"
disable interrupts to set affinity to cpuid.
For example, some results of my rangeley machine
====================
# intrctl list
IRQ CPU#00(+) CPU#02(+) CPU#04(+) CPU#06(+)
CPU#08(+) CPU#10(+) CPU#12(+) CPU#14(+)
3 0* 0 0 0
0 0 0 0 unknown
4 0* 0 0 0
0 0 0 0 unknown
9 0* 0 0 0
0 0 0 0 unknown
18 0* 0 0 0
0 0 0 0 unknown
19 1765* 0 0 0
0 0 0 0 unknown, unknown
20 0* 0 0 0
0 0 0 0 unknown
21 0* 0 0 0
0 0 0 0 unknown
22 0* 0 0 0
0 0 0 0 unknown
23 1022* 0 0 0
0 0 0 0 unknown, unknown
====================
"(+)" mean the CPU accept interrupts. "*" mean the interrupts is set
affinity to the CPU.
"unknown" will be device name if device drivers support showing name.
"unknown, unknown" mean 2 devices share the irq.
====================
# intrctl nointr -c 2
# intrctl list
IRQ CPU#00(+) CPU#02(-) CPU#04(+) CPU#06(+)
CPU#08(+) CPU#10(+) CPU#12(+) CPU#14(+)
3 0* 0 0 0
0 0 0 0 unknown
4 0* 0 0 0
0 0 0 0 unknown
9 0* 0 0 0
0 0 0 0 unknown
18 0* 0 0 0
0 0 0 0 unknown
19 1790* 0 0 0
0 0 0 0 unknown, unknown
20 0* 0 0 0
0 0 0 0 unknown
21 0* 0 0 0
0 0 0 0 unknown
22 0* 0 0 0
0 0 0 0 unknown
23 1449* 0 0 0
0 0 0 0 unknown, unknown
====================
"(-)" mean the CPU doesn't accept interrupts.
====================
# intrctl nointr -c 0
# intrctl list
IRQ CPU#00(-) CPU#02(-) CPU#04(+) CPU#06(+)
CPU#08(+) CPU#10(+) CPU#12(+) CPU#14(+)
3 0 0 0* 0
0 0 0 0 unknown
4 0 0 0* 0
0 0 0 0 unknown
9 0 0 0* 0
0 0 0 0 unknown
18 0 0 0* 0
0 0 0 0 unknown
19 1804 0 5* 0
0 0 0 0 unknown, unknown
20 0 0 0* 0
0 0 0 0 unknown
21 0 0 0* 0
0 0 0 0 unknown
22 0 0 0* 0
0 0 0 0 unknown
23 1507 0 76* 0
0 0 0 0 unknown, unknown
====================
"CPU#00" doesn't accept interrupts, so all interrupt move other cpu which
accept interrupts. "CPU#00"'s IRQ 19 count "1804" mean number of interrupts
which have occured from boot time. The count doesn't increment unless set
affinity the interrupt again.
====================
# intrctl affinity -c 0 -i 23
intrctl: IOC_INTR_AFFINITY: Invalid argument
# intrctl affinity -c 8 -i 23
# intrctl list
IRQ CPU#00(-) CPU#02(-) CPU#04(+) CPU#06(+)
CPU#08(+) CPU#10(+) CPU#12(+) CPU#14(+)
3 0 0 0* 0
0 0 0 0 unknown
4 0 0 0* 0
0 0 0 0 unknown
9 0 0 0* 0
0 0 0 0 unknown
18 0 0 0* 0
0 0 0 0 unknown
19 1804 0 21* 0
0 0 0 0 unknown, unknown
20 0 0 0* 0
0 0 0 0 unknown
21 0 0 0* 0
0 0 0 0 unknown
22 0 0 0* 0
0 0 0 0 unknown
23 1507 0 308 0
17* 0 0 0 unknown, unknown
====================
"CPU#00" doesn't accept interrupts, so "intrctl affinity -c 0 -i 23" failed.
Whereas "CPU#08" accept interrupts, so "intrctl affinity -c 8 -i 23" success
and the interrupt move to "CPU#08".
Could you comment the specification and implementation?
Thanks,
--
//////////////////////////////////////////////////////////////////////
Internet Initiative Japan Inc.
Device Engineering Section,
Core Product Development Department,
Product Division,
Technology Unit
Kengo NAKAHARA <k-nakahara%iij.ad.jp@localhost>