lib/58039: Buffer overflow when writing a SHA512_224 or SHA512_256 digest

[netbsd-bugs%michael-kaufmann.ch@localhost]

>Number:         58039
>Category:       lib
>Synopsis:       Buffer overflow when writing a SHA512_224 or SHA512_256 digest
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Mar 15 10:30:00 +0000 2024
>Originator:     Michael Kaufmann
>Release:        10.0 RC6
>Organization:
>Environment:
NetBSD netbsd.localdomain 10.0_RC6 NetBSD 10.0_RC6 (GENERIC) #0: Tue Mar 12 10:19:02 UTC 2024  mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/amd64/compile/GENERIC amd64
>Description:
When calculating digests with EVP_sha512_224() or EVP_sha512_256(), 64 bytes are written. Applications expect that 28 bytes are written (for EVP_sha512_224) or 32 bytes are written (for EVP_sha512_256). This results in a buffer overflow.

NetBSD uses a patched OpenSSL, with a custom implementation of the SHA512 functions. The original OpenSSL implementation does not have this problem.

Please also see problem report #51333

This bug has been discovered while debugging a curl problem: https://github.com/curl/curl/pull/13070
>How-To-Repeat:
Build and run this program. It crashes on NetBSD, runs fine on other operating systems, e.g. Linux.

gcc -Wall digest.c -lcrypto


#include <stdio.h>
#include <string.h>
#include <openssl/evp.h>

int main()
{
  const char* data = "1";
  size_t length = 1;

  unsigned char digest[32];
  int check_val = 123;

  printf("%d\n", check_val);

  memset(digest, '\0', sizeof(digest));

  EVP_MD_CTX* ctx = EVP_MD_CTX_create();
  if (!ctx) {
    fprintf(stderr, "EVP_MD_CTX_create() failed\n");
    return 1;
  }

  if (!EVP_DigestInit_ex(ctx, EVP_sha512_256(), NULL))
  {
    fprintf(stderr, "EVP_DigestInit_ex() failed\n");
    EVP_MD_CTX_destroy(ctx);
    return 1;
  }

  if (!EVP_DigestUpdate(ctx, data, length))
  {
    fprintf(stderr, "EVP_DigestUpdate() failed\n");
    EVP_MD_CTX_destroy(ctx);
    return 1;
  }

  unsigned int digest_size = 0;
  if (!EVP_DigestFinal_ex(ctx, digest, &digest_size))
  {
    fprintf(stderr, "EVP_DigestFinal_ex() failed\n");
    EVP_MD_CTX_destroy(ctx);
    return 1;
  }

  printf("digest size: %u\n", digest_size);
  printf("%d\n", check_val);

  EVP_MD_CTX_destroy(ctx);

  return 0;
}


Output on NetBSD:

123
digest size: 32
1964291709
[1]   Segmentation fault (core dumped) ./a.out


Output on Linux:

123
digest size: 32
123
>Fix:
Use an unpatched OpenSSL