All of my work so far on i386 MP spinup code are now checked in to the
sommerfeld_i386mp_1 branch.  I'm chasing down a few bugs and
incomplete bits of implementation; it works on my ABIT BP6 (dual
celeron) system but based on some preliminary reports there are a few
issues on other systems.

Significant known bugs:

 - doesn't work if boot cpu is not at apic id 0.
 - support needed for dynamic interrupt allocation isn't quite there yet. 
	(so pcmcia and cardbus won't work right).
 - ioapic without MULTIPROCESSOR doesn't seem to work at the moment

and, of course, I've got a TODO list of little nits about a mile long..

How to try it out:

 a) wait for me to build -current+i386mp GENERIC.MP and INSTALL.MP
kernels and boot floppy images and put them up on ftp.netbsd.org (in
the not too distant future), or

 b) build one of these kernel configs yourself:

Check out -current syssrc as of ~today from cvs or anoncvs, and
then do:

	cd .../sys/arch/i386
	cvs update -rsommerfeld_i386mp_1

which will use that branch in the arch/i386 subtree only.

kernel configs of note:
	GENERIC.MP
		GENERIC with MP options turned on.
	INSTALL.MP
		INSTALL with MP options turned on (to make building a
		miniroot floppy for test boots easier).
	HYDRA
		the development config for my BP6.

If you try this out, send me dmesg output describing success or
failuire.  Include as much detail as you can about the system.

Particularly "interesting" systems include:
	- boxes with more than 2 CPU's
	- boxes with multiple I/O APIC's

If you want to do a custom kernel config with the MP branch code, the
following devices and options are of interest:

cpu* at mainbus? apid ?

	required (well, at least cpu0 is required) even on
	uniprocessor systems.

ioapic* at mainbus? apid ?

	I/O APIC, which fields interrupts from devices and tosses them
	onto the APIC bus (will be required for true MP operation).

	"apid" is "APIC ID", which is the address of the device on the APIC
	interrupt bus.

options MULTIPROCESSOR

	attempt to spin up aux CPU's and maybe do something with them.
	(at this point, not much... :-) )

options LOCKDEBUG

	paranoid lock implementations in an attempt to catch errors..

options MPDEBUG

	debugging of CPU spinup (see arch/i386/i386/mptramp.s; it's
	a simple hack used to debug the second CPU spinup code.

options MPVERBOSE

	spew more details about what it's doing into the logs.

options DEBUG
options DIAGNOSTIC

	you really want these when you're playing with experimental
	stuff.. 

Credits, etc:

 - Ross Harvey for taking the time to answer when I asked "how can I
help make SMP happen".

 - Herb Peyerl for providing the beta PC Weasel 2000 serial console
board which made it possible for me to work on this without going
completely insane (see http://www.realweasel.com).

 - Stefan Grefen for providing the rough draft of the code.

 - and all NetBSD developers for providing a clean, extensible kernel
framework that I could do all this work within.

					- Bill

Here's dmesg output from a boot on my BP6-based system:

NetBSD 1.4T (HYDRA) #4: Mon Feb 21 13:38:37 EST 2000
    sommerfeld@orchard.arlington.ma.us:/d4/vol/smp/arch/i386/compile/HYDRA
total memory = 65088 KB
avail memory = 57404 KB
using 839 buffers containing 3356 KB of memory
BIOS32 rev. 0 found at 0xfb410
mainbus0 (root)
mainbus0: scanning 0x9fc00 to 0x9fff0 for MP signature
mainbus0: scanning 0xf0000 to 0xffff0 for MP signature
mainbus0: MP floating pointer found in bios at 0xf5b90
mainbus0: MP config table at 0xf1400, 292 bytes long
mainbus0: Intel MP Specification (Version 1.1)
mainbus0: MP OEM OEM00000 Product PROD00000000
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: family 6 model 6 step 5
cpu0: Intel Pentium II (Celeron) (686-class)
cpu0: features fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu0: features fbff<PGE,MCA,CMOV>
cpu0: calibrating local timer
cpu0: apic clock running at 66 MHz
cpu0: kstack at 0xc538e000 for 8192 bytes
cpu0: idle pcb at 0xc538e000, idle sp at 0xc538ffa0
cpu1 at mainbus0: apid 1 (application processor)
cpu1: family 6 model 6 step 5
cpu1: Intel Pentium II (Celeron) (686-class)
cpu1: features fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu1: features fbff<PGE,MCA,CMOV>
cpu1: kstack at 0xc5390000 for 8192 bytes
cpu1: idle pcb at 0xc5390000, idle sp at 0xc5391fa0
ioapic0 at mainbus0 apid 2 (I/O APIC)
ioapic0: pa 0xfec00000, virtual wire mode, version 11, 24 pins
ioapic0: int0 attached to ExtINT (type 3<type=3=ExtINT> flags 0<pol=0,trig=0>)
ioapic0: int1 attached to isa0 irq 1 (type 0<type=0> flags 0<pol=0,trig=0>)
ioapic0: int2 attached to isa0 irq 0 (type 0<type=0> flags 0<pol=0,trig=0>)
ioapic0: int3 attached to isa0 irq 3 (type 0<type=0> flags 0<pol=0,trig=0>)
ioapic0: int4 attached to isa0 irq 4 (type 0<type=0> flags 0<pol=0,trig=0>)
ioapic0: int5 attached to isa0 irq 5 (type 0<type=0> flags 0<pol=0,trig=0>)
ioapic0: int6 attached to isa0 irq 6 (type 0<type=0> flags 0<pol=0,trig=0>)
ioapic0: int7 attached to isa0 irq 7 (type 0<type=0> flags 0<pol=0,trig=0>)
ioapic0: int8 attached to isa0 irq 8 (type 0<type=0> flags 5<pol=1=Act Hi,trig=1=Edge>)
ioapic0: int9 attached to isa0 irq 9 (type 0<type=0> flags 0<pol=0,trig=0>)
ioapic0: int12 attached to isa0 irq 12 (type 0<type=0> flags 0<pol=0,trig=0>)
ioapic0: int13 attached to isa0 irq 13 (type 0<type=0> flags 0<pol=0,trig=0>)
ioapic0: int14 attached to isa0 irq 14 (type 0<type=0> flags 0<pol=0,trig=0>)
ioapic0: int15 attached to isa0 irq 15 (type 0<type=0> flags 0<pol=0,trig=0>)
ioapic0: int19 attached to pci0 device 7 INT_A (type 0<type=0> flags f<pol=3=Act Lo,trig=3=Level>)
ioapic0: int18 attached to pci0 device 11 INT_A (type 0<type=0> flags f<pol=3=Act Lo,trig=3=Level>)
ioapic0: int19 attached to pci0 device 17 INT_A (type 0<type=0> flags f<pol=3=Act Lo,trig=3=Level>)
ioapic0: int18 attached to pci0 device 19 INT_A (type 0<type=0> flags f<pol=3=Act Lo,trig=3=Level>)
ioapic0: int18 attached to pci0 device 19 INT_B (type 0<type=0> flags f<pol=3=Act Lo,trig=3=Level>)
ioapic0: int23 attached to SMI (type 2<type=2=SMI> flags 0<pol=0,trig=0>)
local apic: int0 attached to ExtINT (type 3<type=3=ExtINT> flags 0<pol=0,trig=0>)
local apic: int1 attached to NMI (type 1<type=1=NMI> flags 0<pol=0,trig=0>)
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled
pchb0 at pci0 dev 0 function 0
pchb0: Intel 82443BX Host Bridge/Controller (rev. 0x03)
ppb0 at pci0 dev 1 function 0: Intel 82443BX AGP Interface (rev. 0x03)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled
pcib0 at pci0 dev 7 function 0
pcib0: Intel 82371AB PCI-to-ISA Bridge (PIIX4) (rev. 0x02)
pciide0 at pci0 dev 7 function 1: Intel 82371AB IDE controller (PIIX4)
pciide0: bus-master DMA support present
pciide0: primary channel wired to compatibility mode
wd0 at pciide0 channel 0 drive 0: <FUJITSU MPD3084AT>
wd0: drive supports 16-sector pio transfers, lba addressing
wd0: 8063 MB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 16514064 sectors
wd0: 32-bits data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 2
pciide0: primary channel interrupting at irq 14
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2 (using DMA data transfers)
pciide0: secondary channel wired to compatibility mode
pciide0: disabling secondary channel (no drives)
Intel 82371AB USB Host Controller (PIIX4) (USB serial bus, revision 0x01) at pci0 dev 7 function 2 not configured
Intel 82371AB Power Management Controller (PIIX4) (miscellaneous bridge, revision 0x02) at pci0 dev 7 function 3 not configured
tlp0 at pci0 dev 11 function 0: Lite-On 82C169 Ethernet, pass 2.1
tlp0: interrupting at apic 2 int 18 (irq 11)
tlp0: Ethernet address 00:a0:cc:3d:08:6d
lxtphy0 at tlp0 phy 1: LXT970 10/100 media interface, rev. 0
lxtphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
tlp0: supplying EUI64: 00:a0:cc:ff:fe:3d:08:6d
rl0 at pci0 dev 17 function 0: RealTek 8139 Ethernet (id 0x1211)
rl0: interrupting at apic 2 int 19 (irq 10)
rl0: Ethernet address: 00:e0:29:47:b4:85
ukphy0 at rl0 phy 7: Generic IEEE 802.3u media interface
ukphy0: OUI 0x000000, model 0x0000, rev. 0
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
Triones Technologies product 0x0004 (miscellaneous mass storage, revision 0x01) at pci0 dev 19 function 0 not configured
Triones Technologies product 0x0004 (miscellaneous mass storage, revision 0x01) at pci0 dev 19 function 1 not configured
isa0 at pcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
com2 at isa0 port 0x3e8-0x3ef irq 5: ns16550a, working fifo
com2: console
pckbc0 at isa0 port 0x60-0x64
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0
lpt0 at isa0 port 0x378-0x37b irq 7
lpt2 at isa0 port 0x3bc-0x3bf: polled
pcppi0 at isa0 port 0x61
sysbeep0 at pcppi0
npx0 at isa0 port 0xf0-0xff: using exception 16
fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
biomask 0 netmask 0 ttymask 0
cpu0: prelint0 700<vector=0,delmode=7,dest=0> 0<target=0>
cpu0: prelint1 400<vector=0,delmode=4,dest=0> 0<target=0>
cpu0: timer0 300d0<vector=d0,delmode=0,masked,dest=0> 0<target=0>
cpu0: pcint0 10000<vector=0,delmode=0,masked,dest=0> 0<target=0>
cpu0: lint0 10700<vector=0,delmode=7,masked,dest=0> 0<target=0>
cpu0: lint1 400<vector=0,delmode=4,dest=0> 0<target=0>
cpu0: err0 10000<vector=0,delmode=0,masked,dest=0> 0<target=0>
ioapic0: enabling
ioapic0: int0 10700<vector=0,delmode=7,masked,dest=0> 0<target=0>
ioapic0: int1 1a0<vector=a0,delmode=1,dest=0> 0<target=0>
ioapic0: int2 10100<vector=0,delmode=1,masked,dest=0> 0<target=0>
ioapic0: int3 1e1<vector=e1,delmode=1,dest=0> 0<target=0>
ioapic0: int4 1e2<vector=e2,delmode=1,dest=0> 0<target=0>
ioapic0: int5 1e3<vector=e3,delmode=1,dest=0> 0<target=0>
ioapic0: int6 170<vector=70,delmode=1,dest=0> 0<target=0>
ioapic0: int7 1a1<vector=a1,delmode=1,dest=0> 0<target=0>
ioapic0: int8 10100<vector=0,delmode=1,masked,dest=0> 0<target=0>
ioapic0: int9 10100<vector=0,delmode=1,masked,dest=0> 0<target=0>
ioapic0: int10 10000<vector=0,delmode=0,masked,dest=0> 0<target=0>
ioapic0: int11 10000<vector=0,delmode=0,masked,dest=0> 0<target=0>
ioapic0: int12 10100<vector=0,delmode=1,masked,dest=0> 0<target=0>
ioapic0: int13 10100<vector=0,delmode=1,masked,dest=0> 0<target=0>
ioapic0: int14 171<vector=71,delmode=1,dest=0> 0<target=0>
ioapic0: int15 10100<vector=0,delmode=1,masked,dest=0> 0<target=0>
ioapic0: int16 10000<vector=0,delmode=0,masked,dest=0> 0<target=0>
ioapic0: int17 10000<vector=0,delmode=0,masked,dest=0> 0<target=0>
ioapic0: int18 a181<vector=81,delmode=1,actlo,level,dest=0> 0<target=0>
ioapic0: int19 a182<vector=82,delmode=1,actlo,level,dest=0> 0<target=0>
ioapic0: int20 10000<vector=0,delmode=0,masked,dest=0> 0<target=0>
ioapic0: int21 10000<vector=0,delmode=0,masked,dest=0> 0<target=0>
ioapic0: int22 10000<vector=0,delmode=0,masked,dest=0> 0<target=0>
ioapic0: int23 200<vector=0,delmode=2,dest=0> 0<target=0>
IPsec: Initialized Security Association Processing.
boot device: wd0
root on wd0a dumps on wd0b
root file system type: ffs
cpu1: starting
cpu1: init idle stack ptr is 0xc5391fa0
cpu1: prelint0 10000<vector=0,delmode=0,masked,dest=0> 0<target=0>
cpu1: prelint1 10000<vector=0,delmode=0,masked,dest=0> 0<target=0>
cpu1: timer0 10000<vector=0,delmode=0,masked,dest=0> 0<target=0>
cpu1: pcint0 10000<vector=0,delmode=0,masked,dest=0> 0<target=0>
cpu1: lint0 10700<vector=0,delmode=7,masked,dest=0> 0<target=0>
cpu1: lint1 400<vector=0,delmode=4,dest=0> 0<target=0>
cpu1: err0 10000<vector=0,delmode=0,masked,dest=0> 0<target=0>
cpu1: CPU 1 reporting for duty, Sir!
cpu1: stack is 0xc5391f8c
cpu1: sending IPI to cpu 0
cpu1: sending another IPI to cpu 0