lib/60513: lib/curses: wenclose() accepts coordinates one row/column outside the window

["Serhiy Storchaka via gnats" <gnats-admin%NetBSD.org@localhost>]

>Number:         60513
>Category:       lib
>Synopsis:       curses: wenclose() is off by one, so wmouse_trafo() transforms coordinates outside the window
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Jul 28 15:00:01 +0000 2026
>Originator:     Serhiy Storchaka
>Release:        NetBSD 10.1
>Environment:
System: NetBSD 10.1 amd64 (GENERIC), libcurses.so.9.1
>Description:

wenclose() returns true for the row just below a window and for the
column just to its right, so a coordinate outside the window is
reported as enclosed by it. wmouse_trafo() tests the coordinates with
wenclose(), so it transforms and accepts those coordinates too instead
of returning false.

curses_mouse.3 says wenclose() "returns true if the screen relative x
and y co-ordinates are enclosed by the window win, otherwise false",
and the row below a window is not enclosed by it.  ncurses, whose
extension this is, documents the same and returns false there.

The cause is a field with a different meaning in the two
implementations.  The test in libcurses/mouse.c is the one ncurses
uses in ncurses/base/lib_mouse.c:

	/* NetBSD, libcurses/mouse.c */
	if (y < win->begy || y > win->begy + win->maxy ||
	    x < win->begx || x > win->begx + win->maxx)
		return false;

	/* ncurses, ncurses/base/lib_mouse.c */
	result = ((win->_begy <= y && win->_begx <= x &&
		   (win->_begx + win->_maxx) >= x &&
		   (win->_begy + win->_maxy) >= y) ? TRUE : FALSE);

but ncurses' _maxy is the largest valid coordinate ("maximums of x and
y, NOT window size" in its curses.h), while NetBSD's maxy is the
number of lines (newwin.c: "win->maxy = nlines").  So begy + maxy is
the last row in ncurses and one past the last row here.  The rest of
libcurses uses maxy as a count, for example chgat.c "if (x >=
win->maxx || y >= win->maxy)" and move.c "if (x > win->maxx || y >=
win->maxy)"; mouse.c is the only place that reads it the ncurses way.

The code is the same in -current (mouse.c has not changed since the
NULL-pointer checks of 2024-12-23).

>How-To-Repeat:

	/* cc wenclose_off.c -lcurses && TERM=xterm ./a.out */
	#include <curses.h>
	#include <stdio.h>
	int main(void) {
		FILE *f = fopen("/dev/null", "w+");
		SCREEN *sp = newterm("xterm", f, f);
		/* 5 rows x 15 columns at (2,5): rows 2..6, columns 5..19 */
		WINDOW *w = newwin(5, 15, 2, 5);
		printf("wenclose(win, 6, 19) = %d (expected 1)\n", wenclose(w, 6, 19));
		printf("wenclose(win, 7, 19) = %d (expected 0)\n", wenclose(w, 7, 19));
		printf("wenclose(win, 6, 20) = %d (expected 0)\n", wenclose(w, 6, 20));
		endwin(); delscreen(sp);
		return 0;
	}

prints

	wenclose(win, 6, 19) = 1 (expected 1)
	wenclose(win, 7, 19) = 1 (expected 0)
	wenclose(win, 6, 20) = 1 (expected 0)

The same program against ncurses 6.6 prints 1, 0, 0.

Found through Python's curses module, whose test suite checks these
boundaries (test_curses: test_enclose and test_mouse_trafo).

>Fix:

--- a/libcurses/mouse.c
+++ b/libcurses/mouse.c
@@
-	if (y < win->begy || y > win->begy + win->maxy ||
-	    x < win->begx || x > win->begx + win->maxx)
+	if (y < win->begy || y >= win->begy + win->maxy ||
+	    x < win->begx || x >= win->begx + win->maxx)
  		return false;
  	return true;

so that the comparison uses maxy/maxx as the counts they are here.
This should make the reproducer print 1, 0, 0 and wmouse_trafo()
reject the same coordinates; I have not rebuilt libcurses to confirm
it.

>Unformatted: