pkg/60634: digest rmd160 bug for files >= 4GB

["mlelstv%serpens.de@localhost via gnats" <gnats-admin%NetBSD.org@localhost>]

>Number:         60634
>Category:       pkg
>Synopsis:       digest rmd160 bug for files >= 4GB
>Confidential:   no
>Severity:       critical
>Priority:       low
>Responsible:    pkg-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Aug 23 07:20:00 +0000 2026
>Originator:     Michael van Elst
>Release:        NetBSD 11.0_STABLE
>Organization:
>Environment:
System: NetBSD gossam 11.0_STABLE NetBSD 11.0_STABLE (GENERIC) #40: Fri Aug 21 16:29:44 CEST 2026 mlelstv@gossam:/home/netbsd11/obj.amd64/home/netbsd11/src/sys/arch/amd64/compile/GENERIC amd64
Architecture: x86_64
Machine: amd64
>Description:

The pktools/digest implementation of ripemd160 has a flaw that fails to
compute the correct checksum on 64bit systems.

When incrementing the total length value for a hash block, it checks
the lower 32 bits for an overflow to update the upper 32 bits:

         if (context->length[0] + nbytes < context->length[0])
                 context->length[1]++;           /* overflow to msb of length */
         context->length[0] += nbytes;

However, the nbytes value is declared as size_t, which on 64bit systems is 64bit,
so the sum never overflows.

The bug was introduced with rmd160.c 1.7 in 2007 where the nbytes type changed
from uint32_t to size_t.


>How-To-Repeat:
Run digest rmd160 on files larger than 4GB and compare results on 32bit and
64bit architectures.

>Fix:
For example:

Index: rmd160.c
===================================================================
RCS file: /cvsroot/pkgsrc/pkgtools/digest/files/rmd160.c,v
retrieving revision 1.8
diff -p -u -r1.8 rmd160.c
--- rmd160.c    3 Jan 2013 10:20:31 -0000       1.8
+++ rmd160.c    23 Aug 2026 07:08:02 -0000
@@ -365,9 +365,10 @@ RMD160Update(RMD160_CTX *context, const 
        _DIAGASSERT(data != NULL);
 
        /* update length[] */
-       if (context->length[0] + nbytes < context->length[0])
-               context->length[1]++;           /* overflow to msb of length */
+       i = context->length[0];
        context->length[0] += nbytes;
+       if (context->length[0] < i)
+               context->length[1]++;           /* overflow to msb of length */
 
        ZEROIZE(X, sizeof(X));