Re: netbsd-5.1 cpu probe/attach for 11/7xx?

[der Mouse <mouse%Rodents-Montreal.ORG@localhost>]

> But for VMS [on the -11/782], the secondary CPU was scheduled by the
> primary CPU.  It did not run the OS at all.  Instead, [...]

Of course, the real question is, how much of this was imposed by the
hardware and how much is just how VMS happened to use it; just because
the hardware is (almost) symmetric does not necessarily mean software
will use it symmetrically.

>> Certainly the MicroVAX-II supported relatively symmetric
>> multiprocessing; as far as I can recall the only asymmetries there
>> are that the CPU at ID 0 (a) terminates the Qbus, (b) runs at
>> power-up instead of halting and waiting for another CPU to boot it,
>> and (c) gets Qbus device interrupts.
> Well, not really that simple.  There are no IDs for devices on a
> Qbus.

No, but there are for MicroVAX-II CPUs.  There can be up to four of
them per Qbus; for the KA630 and KA620, the ID bits are set by the
connector that on a main CPU runs to the back-panel plate with the
console DE9, speed switch, halt switch, etc.  The usual back-panel
board sets those bits to ID 0, the distinguished CPU, but alternative
boards can set it to any of the other three (as can an ordinary board
with minimal surgery).

> If you wanted to have a second CPU on the Qbus, you need to remove
> the bus termination and bus arbitration from that CPU, since you
> cannot have two bus arbitrators.

I believe using non-0 ID bits does that.  I'd have to check the
KA630/KA620 docs to be sure.

> And as far as I know, the KA630 don't have a way to actually do that
> disabling.

I'm writing from memory here.  I have a scanned copy of
EK-KA630-UG-001, but thanks to monitor trouble, it's difficult for me
to look at it right now; see
ftp.rodents-montreal.org:/mouse/misc/EK-KA630-UG-001/ if you want to
check it out.  But I'm fairly sure just feeding it ID bits other than
00 does what's necessary.  (If you have trouble finding it, I should be
able to look at the files on Monday, or I might even be able to find
the paper original.)

> Also, the uVAX II have the memory on PMI, which is separate from the
> Qbus, so a second CPU cannot access the memory of the primary CPU.

Right.  That's what I mentioned later.  Every CPU has its own pool of
private-interconnect memory, which it may choose to let other CPUs
access - but access by other than the native CPU will involve Qbus
cycles (it just looks like Qbus memory to other CPUs).

> I have actually never seen or heard of a MP KA630,

Well...the KA630 is just the processor.  An MP KA630 does not,
strictly, exist, but, an MP machine may have multiple KA630s as CPUs.

> but if you have some real references, it would be interesting.  I
> looked up the KA620, and it might be that this model was designed
> with the possibility to have several on a single Qbus, but it is a
> bit unclear.

The KA620 is identical to the KA630 except that on the KA630, P0 and P1
page tables live in system virtual space, whereas on the KA620, they
live in physical space.  The only other difference I know of is the way
DEC marketed them - and I worked with them fairly extensively when I
was at McGill.

> I'm also not sure how you dealt with interprocessor communication in
> this case.  I suspect the KA620 used normal Qbus memory, but how
> would one CPU interrupt another one?

The KA620 can use Qbus memory, just as the KA630 can.  But it can also
use private-interconnect memory, just as the KA630 can.  And it has 1MB
onboard, same as the KA630.

Any Qbus initiator can interrupt any KA6[23]0 by a suitable write to
that CPU's IPCR (InterProcessor Communication Register), colloquially
called the doorbell interrupt.  (The IPCR's address depends on the
CPU's ID bits; there are addresses specified for four of them.)

On most CPUs, doorbell interrupts are unreliable; a fraction on the
order of several hundred PPM of interprocessor interrupts get lost.
This is because of a hardware bug.  There's a particular etch run that
crosses the whole board; the resulting capacitance, in conjunction with
the driving impedance, should be insufficient.  That it works even two
or three nines of the time is because DEC routinely overdesigned.  DEC,
to their credit, was able to find and fix this once my colleague was
able to produce convincing evidence of the misbehaviour with a simple
test program, and, as long as a DEC field service organization existed,
you could get the fix (for free, IIRC) if you knew exactly which FCO to
ask for.

When running DEC's multiprocessor thing (VAXELN is the name that comes
to mind), interprocessor doorbells don't need to be reliable; losing a
tiny fraction of them doesn't hurt.  But it did hurt the robot control
system my colleague built.

> KA630 cpu boards exists in ridiculous numbers nowadays, you know... ;-)

I know.  I have at least twice as many KA630s as I have Qbus
backplanes.  If I had two 16MB single-board memory cards I'd be
extremely tempted to put together a two-CPU machine in a BA123.

>>  (Each CPU had its own memory interconnect; the only way to share
>>  memory between CPUs, as far as I know, was over the Qbus.  [...])
> That would also imply DMA access to get to another CPUs memory,
> running through that CPUs mapping registers, and so on.

Right.  That's why you might not have access to all of another CPU's
memory: that CPU's mapping registers might not be configured to.

> It also becomes even more weird, since memory access by one CPU does
> not even go out on the Qbus, so you can only get access to another
> CPUs memory by faking that you are accessing the I/O page of the
> Qbus, which is a 8Kbyte memory in the high end.

I'm fairly sure there was a way to access more than 8K at a time, but
that is getting into things lost to the mists of memory.

> And this has not at all solved the interrupt issues.  If a device
> field an interrupt, all CPUs would see it.  How would you prevent all
> but one CPU to ignore this?

I *think* this too is driven off the ID bits - an ID setting other than
00 disables the Qbus interrupt response circuitry.  Again, I'd have to
check the doc to be sure.

/~\ The ASCII                             Mouse
\ / Ribbon Campaign
 X  Against HTML                mouse%rodents-montreal.org@localhost
/ \ Email!           7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B