lib/50863: libedit el_end() messes up term settings if piped

[a6funxh77z%snkmail.com@localhost]

>Number:         50863
>Category:       lib
>Synopsis:       libedit el_end() messes up term settings if piped
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Feb 27 17:25:00 +0000 2016
>Originator:     John Hein
>Release:        head
>Organization:
none
>Environment:
Noticed with freebsd ports version of libedit (basically 201503 netbsd version).  Reproduced with libedit code from netbsd head on freebsd.

FreeBSD foo.example.com 9.3-STABLE-20150728 FreeBSD 9.3-STABLE-20150728 #8: Fri Jul 31 09:01:31 MDT 2015     foo%example.com@localhost:/usr/obj/usr/src/sys/FOO  i386


Sorry I don't have a netbsd box to use at the moment.  But it's easy to confirm if you do.  And code inspection of head netbsd code seems to show the problem still exists.
>Description:
==================
% cat foo.c
#include <stdio.h>
#include <histedit.h>

static EditLine *el_e = 0;

int
main(int argc, char **argv)
{
        printf("start\n");
        el_e = el_init("foo", stdin, stdout, stderr);
        if (el_e)
                el_end(el_e);
        else
                printf("el_init failed\n");
        printf("end\n");
        printf("end\n");
        return 0;
}
====================

cc foo.c -ledit -o foo && ./foo | cat

The output after el_end() has stair stepped lines (no CR), 'stty -a' shows the baud rate at 50, and on freebsd at least, that messes up things like vi and such.

Looking at the libedit code, it looks like there's a case where code in el_end() tries to restore the original tty settings.  But if the output is not a tty, the original tty settings were never saved.  So it restores invalid termios settings.

The problem is that tty_setup() checks if the output file descriptor is a tty and does a short circuit exit before the termios settings can be saved.  Then el_end(), unaware that the t_or struct is not correctly populated, happily restores the t_or settings that are bogus.
>How-To-Repeat:
See description.
>Fix:
Record original termios settings in tty_setup() even if not isatty().  Or don't try to restore termios settings in tty_end() if tty_setup() never saved them (set a flag or use a pointer for t_or that's null unless the struct is valid or something like that).