On 29 December 2004, Christos Zoulas wrote:

> I think that the problem could be with the non-executable stack and data
> mappings that are new in 2.0. I would add DEBUG_EXEC to my kernel and
> see what that prints. Also you can put some debugging in the trap handler
> for segv in arch/i386/i386/trap.c and print the PC where it dies. Then
> with objdump --dissassemble you can see what it is trying to do.

You were right on the money: I added PROT_EXECUTE to the stack vmcmd and the
SCO COFF executable runs without a hitch.  So I created a function
exec_ibcs2_coff_setup_stack to put in the execsw in place of the regular
exec_setup_stack, and took PROT_EXECUTE back out of the regular setup, so only
ibcs2_coff gets it.  That solves my problem; I don't know if anyone else wants
to dig deeper into what the executable is trying to do and find a better
solution.  My patch (against 2.0 RELEASE sources) is attached.

Never did find a way to get objdump to recognize a SCO COFF executable (that
might have been pilot error). Did however find that the trap is T_PROTFLT with
err 0 and occurs at cs:0x23 eip:0xd4.  Would have tried disassembling that in
ddb if ddb allowed 'print $gdtr' but it wasn't that easy so I gave up. The stack
at that point is ss:1f esp:bfbff928, eax is 0x2000000, ebx is bfbffff0, ds es fs
gs are all 1f, eflags 0x202 and other regs zero in the trapframe.

I still have no answer for the failing ibcs2 elf32 probe, but the executable
I really wanted to run is coff and it runs now so I'll look into the elf
probe some other time.

Thanks for the crucial hint.

-Chap

*** /usr/src/sys/compat/ibcs2/ibcs2_exec.h	Sun Jun 29 17:29:19 2003
--- /usr/src/sys/compat/ibcs2/ibcs2_exec.h	Wed Mar  2 19:11:47 2005
***************
*** 56,61 ****
--- 56,62 ----
  
  struct exec_package;
  int     exec_ibcs2_coff_makecmds __P((struct proc *, struct exec_package *));
+ int	exec_ibcs2_coff_setup_stack __P((struct proc *, struct exec_package *));
  
  /*
   * x.out (XENIX)
*** /usr/src/sys/compat/ibcs2/ibcs2_exec_coff.c	Wed Nov 19 10:46:16 2003
--- /usr/src/sys/compat/ibcs2/ibcs2_exec_coff.c	Wed Mar  2 19:17:59 2005
***************
*** 617,619 ****
--- 617,651 ----
  
  	return 0;
  }
+ 
+ int
+ exec_ibcs2_coff_setup_stack(struct proc *p, struct exec_package *epp)
+ {
+ 	int error;
+ 	struct exec_vmcmd_set *evsp;
+ 	int i;
+ 	
+ 	error = exec_setup_stack( p, epp);
+ 	if ( error ) {
+ 		DPRINTF(("exec_setup_stack unsuccessful\n"));
+ 		return error;
+ 	}
+ 	
+ 	evsp = &epp->ep_vmcmds;
+ 	
+ 	/* look for the stack cmd added by exec_setup_stack: likely the last */
+ 	for ( i = evsp->evs_used ; i --> 0 ; ) {
+ 		struct exec_vmcmd *vcp = evsp->evs_cmds + i;
+ 		if ( vcp->ev_proc != vmcmd_map_zero )
+ 			continue;
+ 		if ( vcp->ev_len  != epp->ep_ssize )
+ 			continue;
+ 		if ( vcp->ev_prot != (VM_PROT_READ | VM_PROT_WRITE) )
+ 			continue;
+ 		vcp->ev_prot |= VM_PROT_EXECUTE;
+ 		return 0;
+ 	}
+ 	
+ 	DPRINTF(("ibcs2_setup_coff_stack: didn't find stack vmcmd?\n"));
+ 	return 0; /* and hope for the best? */
+ }
*** /usr/src/sys/kern/exec_conf.c	Sun Oct 19 02:52:22 2003
--- /usr/src/sys/kern/exec_conf.c	Wed Mar  2 19:12:28 2005
***************
*** 589,595 ****
  	  copyargs,
  	  NULL,
  	  coredump_netbsd,
! 	  exec_setup_stack },
  
  	/* iBCS2 x.out (native word size) */
  	{ XOUT_HDR_SIZE,
--- 589,595 ----
  	  copyargs,
  	  NULL,
  	  coredump_netbsd,
! 	  exec_ibcs2_coff_setup_stack },
  
  	/* iBCS2 x.out (native word size) */
  	{ XOUT_HDR_SIZE,