Subject: tty buffer overflow in ptcwrite()
To: None <>
From: Neelkanth Natu <>
List: tech-kern
Date: 02/16/2002 13:16:17

The following code in ptcwrite() in kern/tty_pty.c is supposed to prevent 
the tty input buffer overflow (in certain cases):

if ((tp->t_rawq.c_cc + tp->t_canq.c_cc) >= TTYHOG - 2 &&
    (tp->t_canq.c_cc > 0 || !ISSET(tp->t_iflag, ICANON))) {
	goto block;

But the ICANON flag is set in tp->t_lflag and not tp->t_iflag. The ICRNL flag
in tp->t_iflag has the same value as the ICANON flag in tp->t_lflag (0x100).
This leads to input buffer overflow as soon as the ICRNL bit is set in 
tp->t_iflag, and there are more than 1024 characters that telnet/ssh want
to write to the pty.

I discovered this problem when using libreadline because it changes the 
terminal settings every time it is begins/finishes reading a line.

The following diff works for me:
< 			   (tp->t_canq.c_cc > 0 || !ISSET(tp->t_iflag, ICANON))) {
> 			   (tp->t_canq.c_cc > 0 || !ISSET(tp->t_lflag, ICANON))) {
< 		      (tp->t_canq.c_cc == 0 && ISSET(tp->t_iflag, ICANON)))))
> 		      (tp->t_canq.c_cc == 0 && ISSET(tp->t_lflag, ICANON)))))

Does that seem to be the right thing to do ?


Do You Yahoo!?
Yahoo! Sports - Coverage of the 2002 Olympic Games