It seems like you are passing the stack limit (512k).Try to increase it (even though I don't understand why openssl would need this amount of stack...)
Same: localhost# limit cputime unlimited filesize unlimited datasize 524288 kbytes stacksize 8192 kbytes coredumpsize unlimited memoryuse unlimited vmemoryuse unlimited descriptors 548 memorylocked unlimited maxproc 148 maxthread 2048 sbsize unlimited localhost# openssl Segmentation fault (core dumped) pkgsrc OpenSSL doesn't compile because of: ld: ./libcrypto.so: undefined reference to `__atomic_is_lock_free' John