[src/trunk]: src/sys/kern tty: Avoid undefined behaviour (left shift of 1 by ...

[nia <nia%NetBSD.org@localhost>]

details:   https://anonhg.NetBSD.org/src/rev/a019a879dfe6
branches:  trunk
changeset: 940530:a019a879dfe6
user:      nia <nia%NetBSD.org@localhost>
date:      Fri Oct 09 09:03:55 2020 +0000

description:
tty: Avoid undefined behaviour (left shift of 1 by 31 places overflows int)

The valid sizes of the tty input and output queues (according to the man page)
are between 1024 and 65536 and input values are converted to a power of two.

The check on the validity of the range is done after the input values are
converted, however, which means that a hostile program can attempt to set
the queue size to a negative value, and cause integer overflow before
the range is validated.

Detected by UBSan

Reported-by: syzbot+521b73969fd233c49e58%syzkaller.appspotmail.com@localhost

diffstat:

 sys/kern/tty.c |  6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diffs (27 lines):

diff -r b786161d30b1 -r a019a879dfe6 sys/kern/tty.c
--- a/sys/kern/tty.c    Fri Oct 09 08:18:01 2020 +0000
+++ b/sys/kern/tty.c    Fri Oct 09 09:03:55 2020 +0000
@@ -1,4 +1,4 @@
-/*     $NetBSD: tty.c,v 1.289 2020/08/26 16:36:32 maxv Exp $   */
+/*     $NetBSD: tty.c,v 1.290 2020/10/09 09:03:55 nia Exp $    */
 
 /*-
  * Copyright (c) 2008, 2020 The NetBSD Foundation, Inc.
@@ -63,7 +63,7 @@
  */
 
 #include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: tty.c,v 1.289 2020/08/26 16:36:32 maxv Exp $");
+__KERNEL_RCSID(0, "$NetBSD: tty.c,v 1.290 2020/10/09 09:03:55 nia Exp $");
 
 #ifdef _KERNEL_OPT
 #include "opt_compat_netbsd.h"
@@ -226,7 +226,7 @@
 static int
 tty_get_qsize(int *qsize, int newsize)
 {
-       if (newsize == 0)
+       if (newsize <= 0)
                return EINVAL;
 
        newsize = 1 << ilog2(newsize);  /* Make it a power of two */