After many hours of find-ing, grepping and cross-referencing, I
finally managed to untangle the NetBSD kernel source tree labyrinth
sufficiently to find the problem preventing the execution of legacy
binaries, and even to implement a dirty hack.  A quick demonstation,
complete with copious debugging output:

    # ./ls                                                    
    Hello_World()!  Trying binary with magic == 0x0000010b
    Yippee!  Found an old style correct-endian ZMAGIC!    
    [               csh             ktrace.out      pagesize        su
    adb             date            ld              passwd          sync
    ar              dd              ln              pr              sync.dis
    as              df              login           ps              tar
    awk             diff            ls              pwd             tee
    cat             du              machine         rcp             test
    cc              e               mail            rm              time
    chfn            echo            make            rmail           tp
    chgrp           ed              mkdir           rmdir           true
    chmod           expr            mt              sed             wall
    chsh            false           mv              sh              who
    cmp             hostid          nice            size            write
    cp              hostname        nm              strip
    cpio            kill            od              stty
    # Hello_World()!  Trying binary with magic == 0x0b019600
    Hello_World()!  Trying binary with magic == 0x0b019600  
    Hello_World()!  Trying binary with magic == 0x0b019680
    Hello_World()!  Trying binary with magic == 0x0b019600
    Hello_World()!  Trying binary with magic == 0x0b019600

In case it hasn't been fixed in a newer release of NetBSD, perhaps
you could clean up my patch and merge it with the official release:
ftp://mariah.narpes.com/users/aw/outgoing/nbsd-vax/netbsd-1.5.23-aw-hack.diff

I had copied the source tree to a PC with a fast SCSI-disk to speed
up searching, but there turned out to be a small version mismatch,
resulting in unnecessarily verbose output from diff(1).  I removed
most of the irrelevancies from the patch mentioned above, except
for the 1.5.2<->1.5.3 difference in the number of arguments taken
by exec_vax1k_prep_anymagic().

According to my possibly erroneous analysis:
* COMPAT_AOUT was not included in the .../vax/conf/* files - I had
  to add it manually.
* However, when I did, the kernel no longer recognised "native" binaries.
* To resolve that, I decided to hack the #defines in kern/exec_conf.c
  to include support for both legacy and native a.out.
* There was some sort of compatibility code in compat/vax1k/vax1k_exec.c -
  including recognition of 43BSD-style MAGIC, but unlike
  compat/aout/aout_exec.c it did not assign &emul_netbsd_aout to
  epp->ep_emul.  Whether that's important, I don't know, but I decided
  to move the COMPAT_43 section from vax1k_exec.c to
  compat/aout/aout_exec.c, so that "maximum" emulation would be applied
  to legacy binaries while vax1k_exec would only recognise new binaries
  with a MID value.
* compat/aout/aout_exec.c originally called the "regular" a.out
  execution code, so I changed that to invoke exec_vax1k_prep_anymagic()
  instead.  I'm not sure whether it was necessary, but it seemed like
  a good idea, and it works!

Regarding the 4K vs 1K page size issue, what's that all about?
Only the option of 512-byte pages is mentioned in any of my VAX
literature (e.g. the Architecture Handbook and "Computer Programming
and Architecture - the VAX" by Levy & Eckhouse).

-aw

On Thu, 2 Mar 2006 ragge@ludd.luth.se wrote:

> It did work back around 1.2-1.3, I used it to run Ultrix 4.5 
> binaries.  But I haven't tried it since then.
> 
> -- Ragge
> 
> > Actually, I don't think that has ever worked.
> > The Ultrix compat stuff is (I think) for the DECstations. Don't think it 
> > has ever been worked over for VAX.
> > 
> > But I'm very unsure about this all. Never tried it myself...
> > 
> >  	Johnny
> > 
> > On Wed, 1 Mar 2006, A. Wik wrote:
> > 
> > > Has anyone tried running 4.3BSD or Ultrix 4.5 binaries
> > > on NetBSD/vax 1.5.x?  Any known problems?
> > >
> > > Although the COMPAT option for 4.3BSD is enabled in my
> > > kernel, all binaries I've tested so far have output some
> > > garbage and exited.  The results were identical to a
> > > preceding attempt with Ultrix compatibility, which
> > > according to kernel config file comments cannot co-exist
> > > with the 4.3 compat option).
> > >
> > > As for the executables, the file(1) utility suggests:
> > >  # file bin/date
> > >  bin/date: 386 demand paged pure executable not stripped
> > >
> > > Manual inspection does not confirm such findings.  For
> > > example, /bin/sync (chosen for its small size) contains
> > > the VAX instruction for a 4.3BSD sync(2) system call -
> > > CHMK 36 - and then immediately checks the carry flag -
> > > with BCS (== BLSSU) to determine whether it failed.
> > >
> > > Abbreviated output of objdump(1) follows:
> > >
> > > # objdump -b binary -m vax -D sync
> > >
> > > sync:     file format binary
> > >
> > > No symbols in "sync".
> > > Disassembly of section .data:
> > >
> > > 00000000 <.data>:
> > >       0:	0b 01 00 00 		crc	$1, $0, $0, $0
> > >       4:	00
> > >       5:	04          		ret
> > >       6:	00          		halt
> > >       7:	00          		halt
> > >       8:	00          		halt
> > >       9:	04          		ret
> > >       a:	00          		halt
> > >       b:	00          		halt
> > >       c:	00          		halt
> > >       d:	00          		halt
> > >       e:	00          		halt
> > >       f:	00          		halt
> > >      10:	f0 00 00 00 		insv	$0, $0, $0, $0
> > >      14:	00
> > > ...
> > >     470:	00          		halt
> > >     471:	00          		halt
> > >     472:	bc 24       		chmk	$36
> > >     474:	1f f2       		blssu	0x468
> > >     476:	04          		ret
> > >
> > >
> > 
> 
>