In article <199711011938.LAA24107@lestat.nas.nasa.gov> you write:
[...]
> Then system calls:
> 
> 	int addinterp(const char *magic, size_t magiclen,
> 	    const char *interp, size_t interplen);
[...]

Some thoughts:

1) DOS .COM files are a problem, because the magics are ambiguous
(really, there is no magic). Perhaps the mechanism should allow "try
these magics, but only if the file has a suffix of <whatever>".

2) Maybe one should also be able to specify a priority when adding a
magic entry, so somewhat ambiguous magics can be tried later, after
those that has a lower false-positive rate.

3) Maybe forget 1 and 2, and have an EXEC_UNKNOWN, that calls
"/usr/libexec/exec_unknown /path/to/original/program [args]"
on anything not recognized by the other EXEC_XXX types, and let the
userland program handle all this junk. (alternate name: EXEC_GENERIC)

4) Completely insane idea: write a 8086 emulator library that uses an
interface identical to the vm86() call, so doscmd and dosemu can run on
other architectures with a minumum of work. (There'd still be some
problems with endianess and type sizes, I'm sure).