bin/42183: sed(1) memory leak

[njoly%pasteur.fr@localhost]

>Number:         42183
>Category:       bin
>Synopsis:       sed(1) memory leak
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    bin-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Oct 14 13:10:00 +0000 2009
>Originator:     Nicolas Joly
>Release:        NetBSD 5.99.20
>Organization:
Intitut Pasteur
>Environment:
System: NetBSD lanfeust.sis.pasteur.fr 5.99.20 NetBSD 5.99.20 (LANFEUST) #2: 
Tue Oct 13 20:49:48 CEST 2009 
njoly%lanfeust.sis.pasteur.fr@localhost:/local/src/NetBSD/obj.amd64/sys/arch/amd64/compile/LANFEUST
 amd64
Architecture: x86_64
Machine: amd64
>Description:
While processing a 41GB file with sed, i noticed that the process was eating
memory up top 4GB when i stoppped it.

A further analysis show that it leaks 128 bytes for each processed line.

njoly@lanfeust [~]> printf "" | mleaks sed '' >/dev/null
sed: found 3 unallocated objects (56 bytes)
        from sed:err+0x109 ... 16 bytes
        from sed:err+0x109 ... 16 bytes
        from sed:err+0x109 ... 24 bytes
njoly@lanfeust [~]> printf "foo\n" | mleaks sed '' >/dev/null 
sed: found 5 unallocated objects (1213 bytes)
        from sed:err+0x109 ... 1029 bytes
        from libc.so.12:__getdelim+0x7f ... 128 bytes
        from sed:err+0x109 ... 16 bytes
        from sed:err+0x109 ... 16 bytes
        from sed:err+0x109 ... 24 bytes
njoly@lanfeust [~]> printf "foo\nfoo\n" | mleaks sed '' >/dev/null
sed: found 6 unallocated objects (1341 bytes)
        from libc.so.12:__getdelim+0x7f ... 128 bytes
        from sed:err+0x109 ... 1029 bytes
        from libc.so.12:__getdelim+0x7f ... 128 bytes
        from sed:err+0x109 ... 16 bytes
        from sed:err+0x109 ... 16 bytes
        from sed:err+0x109 ... 24 bytes
And so on ...

The leaks comes from the fgetstr() call in sed sources :

#0  __getdelim (buf=0x7f7fffffda30, buflen=0x7f7fffffda28, sep=10, 
    fp=0x7f7ffdb13cc0) at /local/src/NetBSD/src/lib/libc/stdio/getdelim.c:58
#1  0x00007f7ffd8a663b in __fgetstr (fp=0x7f7ffdb13cc0, lenp=0x7f7fffffda58, 
    sep=10) at /local/src/NetBSD/src/lib/libc/stdio/fgetstr.c:60
#2  0x00000000004029ea in mf_fgets (sp=0x607c20, spflag=REPLACE)
    at /local/src/NetBSD/src/usr.bin/sed/main.c:336
#3  0x0000000000403721 in process ()
    at /local/src/NetBSD/src/usr.bin/sed/process.c:137
#4  0x0000000000403071 in main (argc=3, argv=0x7f7fffffdb38)
    at /local/src/NetBSD/src/usr.bin/sed/main.c:197

>How-To-Repeat:
Try to run sed on a very big file ... and watch the process memory increase.
>Fix:
n/a