While trying to make the procfs work for use by a debugger, I'm now somewhat stuck with a problem regarding the masked SIGTRAP handling.

In short, the problem is that you cannot set a breakpoint in the signal
handler for signal SIGTRAP.

To see the problem, compile the following little program:

#include <stdio.h>
#include <signal.h>

void catch(sig)
{
        printf("signal %d caught\n",sig);
}

main()
{
        signal(SIGINT,&catch);
        signal(SIGTRAP,&catch);
        while (1);
}

Then start gdb with it, setting a breakpoint in catch. After starting the
program, interrupt it with e.g. ^C. Continuing from here with `signal 2'
several times works as expected. But if you try to continue with `signal 5'
you will get something like SIGILL, SIGBUS or some such.

The problem is that the TRAP signal is masked within the signal handler, and
even the debugger isn't told about the arrival of this signal on execution
of the break instruction. The breakpoint is simply ignored by the system
and the program continued after the int3 instruction. And since the int3
was patched over another instruction, the remainder of that instruction
is most likely something illegal.

BTW, other OSs have their problem with this, too. On SunOS 4.1.1, the
inferior process dumps core under these circumstances.

IMHO, we should try to find a solution that works similar to the handling
of other signals like SIGINT above. In the current kernel this would mean
to even deliver masked signals to the debugger like is done with ignored
signals.

Any ideas?
--
ws@TooLs.DE     (Wolfgang Solfrank, TooLs GmbH) +49-228-985800

------------------------------------------------------------------------------