I am running a machine at home with 40 megs of RAM, 2 IDE drives with a 
   total of 1.4 gigs storage, a nameless vlb svga controller with an S868<?> 
   chip, promise eide controller, and a Nx586-100 motherboard.

   Will either netbsd or freebsd:
   a) run on the Nx586?

I'm not aware of any incompatibilities that should prevent NetBSD from
working.  However, I've not actually seen a machine with a NexGen
chip, so I can't attest that it actually does.

   b) gain any advantage on the Nx586, over the Pentium, by ducking the CISC 
   code altogether and taking advantage of the RISC architecture?

Not yet.