Having spent a weekend fiddling with interrupts, then a few nights tidying it up, I've come up with new irq handling for cats.  It's loosely based on the evbarm/iq80310 by Jason, but adjusted for having proper hardware masking, and doing the priority ordering right.

I've uploaded the diff and GENERIC cats kernel to:
ftp://ftp.netbsd.org/pub/NetBSD/arch/cats/newirq/

Main changes are:
* Only a stub asm irq function, it calls into a C irq dispatch routine, basically minimal .S, I'm hoping that we might be able reuse this code more widely accross the arm ports.
* Generic soft interrupts, generic soft interrupts will also use the hardware supported soft interrupt line, soft net, clock and serial use fake interrupts.
* interrupts are handled in priority order (it was in lowest irq to highest irq number) soft interrupts will also be handled as we drop down the spl levels, eg a softserial will get handled over a bio or net, if one is set, IE soft serial really will take priority over bio and net.
* tidied up isa_machdep.c (only the cats version, hence the discussion of merging)
* evcnt's are used for interrupt stats, no IRQSTATS define any more (note that systat vm doesn't do evcnt, but vmstat -i does, or perhaps I'm missing something)
* All extra footbridge intrs (aborts, sdram errors, parity errors) are no at IPL_HIGH, not IPL_NONE, I view them as critical to get through, not well we might care eventually, is this the wrong view to take?

Things to do:
* tidy up whitespace, it does need tidying.
* sort out what's in irqhandler.h and intr.h
* Anything any one else comes up with

Things to do post checkin:
* inline versions of the spl* code.
* inline the irq_mask setting, could easily be an inlined function
* tidy up local variables, such that they are static
* implement CLKF_BASEPRI, although I'm not sure the description is right in cpu.h, basically it says we were at spl0 when we took the interrupt.  Whereas I feel it should be were we at an spl < IPL_SOFTCLOCK and are there no other hardware or software interrupts pending.
* fix isa interrupt handling, so IPL's means something for isa bus. (really must must must fix this)
* merge in netwinder (actually the only deviation is in isa_machdep.c where the irq line is different)
* probably more that my tired brain can't remember

Things this didn't do:
* fix the UDMA issue with multiple disks. (which I had hoped it would, I'm wondering if that needs the isa irq handling reworking...)

Things to be aware of if you do look over it (bear in mind comments may not be accurate):
* footbridge_imask is the bitmasks of interrupts that can be enabled at that IPL level, not the interrupts to mask.
* intr_enabled is the mask of all interrupts that we have handlers for (except for soft interrupts)
* current_ipl is the current ipl level, not a mask or similair, it's only written to with interrupts disabled
* higher ipl's block all lower ipl levels.
* when handling an IRQ we go to the highest IPL level for that irq
* soft interrupts are handled in the same was as irq's, IE they have a irq dispatch routine.
* the loop in irq_dispatch is cunning, in that it merges in soft interrupts that can be handled at the previous ipl level, and will deal with them now, this means that all soft interrupts that can be dealt with are, then we can return to the previous level, IE we never return to an interrupt handler with any higher priority interrupts pending.  This makes sure that SOFTSERIAL is handled before IPL_BIO and IPL_NET.

Anyway all sensible suggestions comments etc appreciated.

Cheers,
Chris