Re: lib/58039: Buffer overflow when writing a SHA512_224 or SHA512_256 digest

[Taylor R Campbell <riastradh%NetBSD.org@localhost>]

Thanks for the report!

Obviously, the hole we've dug ourselves into with the OpenSSL SHA-2
API is a bad state of affairs.

Unfortunately, it is not as easy as simply not patching openssl.

There is a namespace collision between NetBSD's libc and (unpatched)
OpenSSL over names like `SHA512_Init' and `SHA512_Final', which are
declared with different SHA512_CTX types of different sizes.

And we have ABI compatibility requirements that prevent us from just
dropping the symbols from libc, or, worse, pulling the definitions out
of OpenSSL instead -- applications previously using the libc symbols
and NetBSD's SHA512_CTX would suddenly get buffer overruns, because
OpenSSL's SHA512_CTX is larger!

Perhaps we could patch OpenSSL _just to rename the symbols_, and make
sure OpenSSL's libcrypto is never using the libc symbols -- that way
_new_ programs which include <openssl/sha.h> will get the OpenSSL
symbols, and _old_ programs will still get the libc symbols, and
OpenSSL will use its own internal API internally.  And maybe that will
reduce our maintenance burden.

But past attempts to dig out of this hole have met with various kinds
of gnarly failure, and OpenSSL is ditching the easy-to-use C APIs for
things like SHA-2 and AES anyway.  So for now I've just put another
band-aid on the mess -- and added some automatic tests to catch the
problem later.