Re: lib/58039: Buffer overflow when writing a SHA512_224 or SHA512_256 digest

[Taylor R Campbell <riastradh%NetBSD.org@localhost>]

The following reply was made to PR lib/58039; it has been noted by GNATS.

From: Taylor R Campbell <riastradh%NetBSD.org@localhost>
To: Michael Kaufmann <netbsd-bugs%michael-kaufmann.ch@localhost>
Cc: gnats-bugs%NetBSD.org@localhost, netbsd-bugs%NetBSD.org@localhost
Subject: Re: lib/58039: Buffer overflow when writing a SHA512_224 or SHA512_256 digest
Date: Fri, 15 Mar 2024 19:57:30 +0000

 Thanks for the report!
 
 Obviously, the hole we've dug ourselves into with the OpenSSL SHA-2
 API is a bad state of affairs.
 
 Unfortunately, it is not as easy as simply not patching openssl.
 
 There is a namespace collision between NetBSD's libc and (unpatched)
 OpenSSL over names like `SHA512_Init' and `SHA512_Final', which are
 declared with different SHA512_CTX types of different sizes.
 
 And we have ABI compatibility requirements that prevent us from just
 dropping the symbols from libc, or, worse, pulling the definitions out
 of OpenSSL instead -- applications previously using the libc symbols
 and NetBSD's SHA512_CTX would suddenly get buffer overruns, because
 OpenSSL's SHA512_CTX is larger!
 
 Perhaps we could patch OpenSSL _just to rename the symbols_, and make
 sure OpenSSL's libcrypto is never using the libc symbols -- that way
 _new_ programs which include <openssl/sha.h> will get the OpenSSL
 symbols, and _old_ programs will still get the libc symbols, and
 OpenSSL will use its own internal API internally.  And maybe that will
 reduce our maintenance burden.
 
 But past attempts to dig out of this hole have met with various kinds
 of gnarly failure, and OpenSSL is ditching the easy-to-use C APIs for
 things like SHA-2 and AES anyway.  So for now I've just put another
 band-aid on the mess -- and added some automatic tests to catch the
 problem later.