Nested functions [was Re: valgrind]

[Mouse <mouse%Rodents-Montreal.ORG@localhost>]

>> Can't you?  Does C require function pointers to have the same type,
>> or compatible structure, as data pointers?
> No, I don't think that it does.

Correct.

> You could have different sizes for those.

Indeed, you can have different sizes for pointers to different object
types, too.  I _think_ pointers to different function types can have
different sizes, but I'm less certain of that.  (There would be little
point, since all function pointer types have to have the same
information content; see below.)

> However there are, I fear, too many programs that somewhere convert a
> function pointer to (void *) and at that point things break.

That is a bug.  It always has been.  Such code is broken and it
deserves to be rednered _obviously_ broken so it can get fixed.  I can
understand commercial compiler vendors' reluctance to "break customer
code" even if the reality is more "pointing out where customer code was
already broken and just historically getting away with it".  But I have
more trouble understanding it for things like gcc.

But then, there is a lot of confusion around code portability.  There's
at least one relatively popular open-source project - SQLite, fuzzy
memory says - that has a FAQ list entry that goes something like
"$PROJECT provokes these warnings!" with a response that used to read
like "we test it heavily, the code is fine", not understanding that the
warnings are not so much about the code generated today on today's
machines, which they are correct that testing can address; it is about
tomorrow's architecture and/or tomorrow's new compiler release, or new
compiler, getting the code correct so it will work there too.

> And there is not really a "generic function pointer type" that you
> could sensibly use instead, I think.

There is, actually.  Any function pointer type will do.  Any function
pointer can be cast to any other function pointer type and back without
change - at least as of C99; "A pointer to a function of one type may
be converted to a pointer to a function of another type and back again;
the result shall compare equal to the original pointer.".  What you
can't do is call through the pointer when it points to the wrong
function type: "If a converted pointer is used to call a function whose
type is not compatible with the pointed-to type, the behavior is
undefined.".

> Possibly, a trampoline could be created on the heap, and then made
> executable and un-writable. Maybe that's considered too complicated /
> system dependent / expensive by gcc?

(1) it is ugly for the runtime to be mallocing behind the scenes, (2)
this breaks in the presence of longjmp (it is hard to stop it from
leaking trampolines when longjmping through a stack frame that created
a trampoline), and (3) it is difficult to use safely in the presence of
signal pointers or threads.  Oh, and (4) yes, changing memory
protection is system-dependent and expensive, and there are even some
Harvard(ish) architectures on which generating new executable code at
runtime is architecturally impossible.  (Few to none of them are
targeted by gcc, I suspect.)

(2) could be addressed if someone were to design an unwind-protect for
longjmp (which arguably should have existed all along), but that brings
it back to the "gcc wants to be compatible with existing systems"
issue; remember, back when gcc arose it usually wasn't the system
compiler, so it had to be compatible.  Arguably, now that gcc at least
sometimes _is_ the system compiler, it would make sense for it to grow
a way to handle nested functions better.  But I suspect there is
comparatively little will to do that in the gcc crowd, or it would have
happened long since.  And the heap techniques still run into (3); I
think fat function pointers is the rightest way to do it when
compatibility with a preexisting ABI is not a concern.

To bring this back to NetBSD, I ran into (4) myself back sometime in
2000-2005ish, I think it was.  There were two issues I ran into; I
can't recall which one was first.  One was that some new NetBSD release
brought in totally non-executable stack on at least one of the
architectures I ran, making it impossible to use gcc nested functions
at all.  That I had to fix in the kernel.  The other was that gcc's
configuration made at least one syscall per trampoline generated.  The
results were correct, but it was intolerably slow for at least one
program I cared about (I think it was a search program with a nested
function pointer being generated in a relatively deeply nested
routine).  I forget what I did there - just made the whole stack
executable, maybe?

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		mouse%rodents-montreal.org@localhost
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B