First of all, are you using the recent com driver, or the old version?
If the new version, are you actually seeing `silo' overflows or `ibuf'
overflows?

There are many things that can cause interrupts to be delayed for
short periods.  Combined with the overhead of actually entering the
interrupt routine and calling up to the line discipline input routine,
the total latency before the next character is read can easily be too
high for an unbuffered serial port.  There are a few ways to partially
fix this:

1) Add an extra layer of buffering, to shorten the path inside the
interrupt handler.  This has been done.

2) Give tty interrupts a higher priority.  I was planning to do this
soon.  You could go further and (almost) never allow the lower half of
the interrupt handler to be blocked.  This would give you close to the
minimum possible latency.

3) Modify the line discipline interface to allow passing up a larger
number of characters at once.  For things like SLIP and PPP, this
could significantly reduce overhead by eliminating function calls and
allowing the line discipline to have tighter loops for several things.
This would help prevent overflowing the secondary buffer.

I would guess that items 2 and 3 could be implemented in a weekend.  I
don't currently have a weekend free, though.  (hint, hint)

The FreeBSD code mostly does item 2.  It goes to a slight extreme on
item 3, by inlining part of the generic tty interface into the serial
driver.  While this probably improves performance, it's not really
acceptable from an architectural viewpoint.