> In special case, when I give it '\(.\)\1\1\1' (any 4 sequential
> characters), its size will be increasing to its limit (which
> can be monitord by top(1)).

At a first glance, following free()s are missing.

enami.
Index: engine.c
===================================================================
RCS file: /cvsroot/basesrc/lib/libc/regex/engine.c,v
retrieving revision 1.12
diff -u -r1.12 engine.c
--- engine.c	1999/09/16 11:45:20	1.12
+++ engine.c	2001/12/10 06:17:25
@@ -202,6 +202,10 @@
 	for (;;) {
 		endp = fast(m, start, stop, gf, gl);
 		if (endp == NULL) {		/* a miss */
+			if (m->pmatch != NULL)
+				free(m->pmatch);
+			if (m->lastpos != NULL)
+				free(m->lastpos);
 			STATETEARDOWN(m);
 			return(REG_NOMATCH);
 		}