RFC: NUMA support

To: tech-kern%netbsd.org@localhost
Subject: RFC: NUMA support
From: Christoph Egger <Christoph_Egger%gmx.de@localhost>
Date: Mon, 10 Nov 2008 17:11:37 +0200
Hi!

I started to work on NUMA support. First step is to set up the
topology.

It does this by scanning the ACPI SRAT table.
If no ACPI SRAT table is present or if you boot w/ ACPI disabled,
a one node NUMA system is faked.

The boot code also utilizes the ACPI MADT table to get more
information so far possible.

I showed rmind@ my patch so far. I share his opinion, that
it needs some more thoughts on the MI API side.
Nonetheless, it's a start.

The next two items are to write a numactl(8) utility and
to utilize the BIOS e820 memory map for more detailed
and accurate information on the NUMA memory
layout -  the memory holes in particular.

The dmesg snippet on a four-socket machine looks like this:

NUMA: SRAT table found
NUMA: SLIT table not found
ioapic0 at mainbus0 apid 0
ioapic1 at mainbus0 apid 1
numa0 at mainbus0
cpu0 at numa0 apic 4 (BP): AMD 686-class, 2300 MHz, id 0x100f40
cpu1 at numa0 apic 5 (AP): AMD 686-class, 2300 MHz, id 0x100f40
cpu2 at numa0 apic 6 (AP): AMD 686-class, 2300 MHz, id 0x100f40
cpu3 at numa0 apic 7 (AP): AMD 686-class, 2300 MHz, id 0x100f40
numa0: memory: 0x0 - 0xa0000 (0xa0000, physical, raw, raw)
numa0: memory: 0x100000 - 0x40000000 (0x3ff00000, physical, raw, raw)
numa1 at mainbus0
cpu4 at numa1 apic 8 (AP): AMD 686-class, 2300 MHz, id 0x100f40
cpu5 at numa1 apic 9 (AP): AMD 686-class, 2300 MHz, id 0x100f40
cpu6 at numa1 apic 10 (AP): AMD 686-class, 2300 MHz, id 0x100f40
cpu7 at numa1 apic 11 (AP): AMD 686-class, 2300 MHz, id 0x100f40
numa1: memory: 0x40000000 - 0x80000000 (0x40000000, physical, raw, raw)
numa2 at mainbus0
cpu8 at numa2 apic 12 (AP): AMD 686-class, 2300 MHz, id 0x100f40
cpu9 at numa2 apic 13 (AP): AMD 686-class, 2300 MHz, id 0x100f40
cpu10 at numa2 apic 14 (AP): AMD 686-class, 2300 MHz, id 0x100f40
cpu11 at numa2 apic 15 (AP): AMD 686-class, 2300 MHz, id 0x100f40
numa2: memory: 0x80000000 - 0xc0000000 (0x40000000, physical, raw, raw)
numa3 at mainbus0
cpu12 at numa3 apic 16 (AP): AMD 686-class, 2300 MHz, id 0x100f40
cpu13 at numa3 apic 17 (AP): AMD 686-class, 2300 MHz, id 0x100f40
cpu14 at numa3 apic 18 (AP): AMD 686-class, 2300 MHz, id 0x100f40
cpu15 at numa3 apic 19 (AP): AMD 686-class, 2300 MHz, id 0x100f40
numa3: memory: 0xc0000000 - 0xd8000000 (0x18000000, physical, raw, raw)
numa3: memory: 0x100000000 - 0x128000000 (0x28000000, physical, raw, raw)

I can also suspend and resume a full node:

# cpuctl list
Num  HwId Unbound LWPs Interrupts     Last change
---- ---- ------------ -------------- ----------------------------
0    0    online       intr           Wed Nov  5 01:55:47 2008
1    1    online       intr           Wed Nov  5 01:55:47 2008
2    2    online       intr           Wed Nov  5 01:55:47 2008
3    3    online       intr           Wed Nov  5 01:55:47 2008
4    4    online       intr           Wed Nov  5 01:55:47 2008
5    5    online       intr           Wed Nov  5 01:55:47 2008
6    6    online       intr           Wed Nov  5 01:55:47 2008
7    7    online       intr           Wed Nov  5 01:55:47 2008
8    8    online       intr           Wed Nov  5 01:55:47 2008
9    9    online       intr           Wed Nov  5 01:55:47 2008
10   a    online       intr           Wed Nov  5 01:55:47 2008
11   b    online       intr           Wed Nov  5 01:55:47 2008
12   c    online       intr           Wed Nov  5 01:55:47 2008
13   d    online       intr           Wed Nov  5 01:55:47 2008
14   e    online       intr           Wed Nov  5 01:55:47 2008
15   f    online       intr           Wed Nov  5 01:55:47 2008

# drvctl -l mainbus0
mainbus0 ioapic0
mainbus0 ioapic1
mainbus0 numa0
mainbus0 numa1
mainbus0 numa2
mainbus0 numa3
mainbus0 acpi0
mainbus0 pci0
mainbus0 pci8

# drvctl -S numa2
# cpuctl list
Num  HwId Unbound LWPs Interrupts     Last change
---- ---- ------------ -------------- ----------------------------
0    0    online       intr           Wed Nov  5 01:55:47 2008
1    1    online       intr           Wed Nov  5 01:55:47 2008
2    2    online       intr           Wed Nov  5 01:55:47 2008
3    3    online       intr           Wed Nov  5 01:55:47 2008
4    4    online       intr           Wed Nov  5 01:55:47 2008
5    5    online       intr           Wed Nov  5 01:55:47 2008
6    6    online       intr           Wed Nov  5 01:55:47 2008
7    7    online       intr           Wed Nov  5 01:55:47 2008
8    8    offline      intr           Wed Nov  5 02:03:57 2008
9    9    offline      intr           Wed Nov  5 02:03:57 2008
10   a    offline      intr           Wed Nov  5 02:03:57 2008
11   b    offline      intr           Wed Nov  5 02:03:57 2008
12   c    online       intr           Wed Nov  5 01:55:47 2008
13   d    online       intr           Wed Nov  5 01:55:47 2008
14   e    online       intr           Wed Nov  5 01:55:47 2008
15   f    online       intr           Wed Nov  5 01:55:47 2008

# drvctl -Q numa2
# cpuctl list
Num  HwId Unbound LWPs Interrupts     Last change
---- ---- ------------ -------------- ----------------------------
0    0    online       intr           Wed Nov  5 01:55:47 2008
1    1    online       intr           Wed Nov  5 01:55:47 2008
2    2    online       intr           Wed Nov  5 01:55:47 2008
3    3    online       intr           Wed Nov  5 01:55:47 2008
4    4    online       intr           Wed Nov  5 01:55:47 2008
5    5    online       intr           Wed Nov  5 01:55:47 2008
6    6    online       intr           Wed Nov  5 01:55:47 2008
7    7    online       intr           Wed Nov  5 01:55:47 2008
8    8    online       intr           Wed Nov  5 02:03:59 2008
9    9    online       intr           Wed Nov  5 02:03:59 2008
10   a    online       intr           Wed Nov  5 02:03:59 2008
11   b    online       intr           Wed Nov  5 02:03:59 2008
12   c    online       intr           Wed Nov  5 01:55:47 2008
13   d    online       intr           Wed Nov  5 01:55:47 2008
14   e    online       intr           Wed Nov  5 01:55:47 2008
15   f    online       intr           Wed Nov  5 01:55:47 2008


Playing around, I found a bug in respect to cpu0:

# drvctl -S numa0
# cpuctl list
Num  HwId Unbound LWPs Interrupts     Last change
---- ---- ------------ -------------- ----------------------------
0    0    online       intr           Wed Nov  5 01:55:47 2008
1    1    offline      intr           Wed Nov  5 02:27:01 2008
2    2    offline      intr           Wed Nov  5 02:27:01 2008
3    3    offline      intr           Wed Nov  5 02:27:01 2008
4    4    online       intr           Wed Nov  5 01:55:47 2008
5    5    online       intr           Wed Nov  5 01:55:47 2008
6    6    online       intr           Wed Nov  5 01:55:47 2008
7    7    online       intr           Wed Nov  5 01:55:47 2008
8    8    online       intr           Wed Nov  5 02:03:59 2008
9    9    online       intr           Wed Nov  5 02:03:59 2008
10   a    online       intr           Wed Nov  5 02:03:59 2008
11   b    online       intr           Wed Nov  5 02:03:59 2008
12   c    online       intr           Wed Nov  5 01:55:47 2008
13   d    online       intr           Wed Nov  5 01:55:47 2008
14   e    online       intr           Wed Nov  5 01:55:47 2008
15   f    online       intr           Wed Nov  5 01:55:47 2008

# cpuctl offline 0
# cpuctl list      
Num  HwId Unbound LWPs Interrupts     Last change
---- ---- ------------ -------------- ----------------------------
0    0    offline      intr           Wed Nov  5 02:27:09 2008
1    1    offline      intr           Wed Nov  5 02:27:01 2008
2    2    offline      intr           Wed Nov  5 02:27:01 2008
3    3    offline      intr           Wed Nov  5 02:27:01 2008
4    4    online       intr           Wed Nov  5 01:55:47 2008
5    5    online       intr           Wed Nov  5 01:55:47 2008
6    6    online       intr           Wed Nov  5 01:55:47 2008
7    7    online       intr           Wed Nov  5 01:55:47 2008
8    8    online       intr           Wed Nov  5 02:03:59 2008
9    9    online       intr           Wed Nov  5 02:03:59 2008
10   a    online       intr           Wed Nov  5 02:03:59 2008
11   b    online       intr           Wed Nov  5 02:03:59 2008
12   c    online       intr           Wed Nov  5 01:55:47 2008
13   d    online       intr           Wed Nov  5 01:55:47 2008
14   e    online       intr           Wed Nov  5 01:55:47 2008
15   f    online       intr           Wed Nov  5 01:55:47 2008

# drvctl -Q numa0
# cpuctl list     
Num  HwId Unbound LWPs Interrupts     Last change
---- ---- ------------ -------------- ----------------------------
0    0    offline      intr           Wed Nov  5 02:27:09 2008
1    1    online       intr           Wed Nov  5 02:27:15 2008
2    2    online       intr           Wed Nov  5 02:27:15 2008
3    3    online       intr           Wed Nov  5 02:27:15 2008
4    4    online       intr           Wed Nov  5 01:55:47 2008
5    5    online       intr           Wed Nov  5 01:55:47 2008
6    6    online       intr           Wed Nov  5 01:55:47 2008
7    7    online       intr           Wed Nov  5 01:55:47 2008
8    8    online       intr           Wed Nov  5 02:03:59 2008
9    9    online       intr           Wed Nov  5 02:03:59 2008
10   a    online       intr           Wed Nov  5 02:03:59 2008
11   b    online       intr           Wed Nov  5 02:03:59 2008
12   c    online       intr           Wed Nov  5 01:55:47 2008
13   d    online       intr           Wed Nov  5 01:55:47 2008
14   e    online       intr           Wed Nov  5 01:55:47 2008
15   f    online       intr           Wed Nov  5 01:55:47 2008

# cpuctl online 0
# cpuctl list     
Num  HwId Unbound LWPs Interrupts     Last change
---- ---- ------------ -------------- ----------------------------
0    0    online       intr           Wed Nov  5 02:27:22 2008
1    1    online       intr           Wed Nov  5 02:27:15 2008
2    2    online       intr           Wed Nov  5 02:27:15 2008
3    3    online       intr           Wed Nov  5 02:27:15 2008
4    4    online       intr           Wed Nov  5 01:55:47 2008
5    5    online       intr           Wed Nov  5 01:55:47 2008
6    6    online       intr           Wed Nov  5 01:55:47 2008
7    7    online       intr           Wed Nov  5 01:55:47 2008
8    8    online       intr           Wed Nov  5 02:03:59 2008
9    9    online       intr           Wed Nov  5 02:03:59 2008
10   a    online       intr           Wed Nov  5 02:03:59 2008
11   b    online       intr           Wed Nov  5 02:03:59 2008
12   c    online       intr           Wed Nov  5 01:55:47 2008
13   d    online       intr           Wed Nov  5 01:55:47 2008
14   e    online       intr           Wed Nov  5 01:55:47 2008
15   f    online       intr           Wed Nov  5 01:55:47 2008


The NUMA patch is here:
http://www.netbsd.org/~cegger/numa.diff

A note, this patch breaks compiling Xen2 kernels. But since Xen2 is going
to be removed in -current, I don't care.
(To be honest, my Xen2 "fix" is this: 
http://www.netbsd.org/~cegger/rm_xen2.diff)


Christoph
Follow-Ups:
- Re: RFC: NUMA support
  - From: David Young
- Re: RFC: NUMA support
  - From: Andrew Doran
- Re: RFC: NUMA support
  - From: Rhialto
- Re: RFC: NUMA support
  - From: Manuel Bouyer
Prev by Date: Re: Proposal for new syscalls: getrusagex and waitx
Next by Date: -current panic on amd64
Previous by Thread: Questions regarding acpi_tz(4)
Next by Thread: Re: RFC: NUMA support
Indexes:
Home | Main Index | Thread Index | Old Index