Re: gzimg hashes

[Brook Milligan <brook%biology.nmsu.edu@localhost>]

> On Jul 5, 2026, at 20:12, David H. Gutteridge <david%gutteridge.ca@localhost> wrote:
> 
> On Sun, 5 Jul 2026 at 17:42:17 -0600, Brook Milligan wrote:
>> I notice that the hashes generated for release files in the gzimg
>> directory cover the uncompressed files, not the compressed files
>> actually distributed.  This seems to be a mistake.
>> 
>> The fix is clearly to reorder the hash generation relative to the
>> compression, but I’m checking here to make sure I’m not missing
>> something.
>> 
>> Cheers,
>> Brook
> 
> Yes, that's one of the issues raised in PR 59195 (and its duplicate,
> PR 60302), e.g.,
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail-index.netbsd.org%2Fnetbsd-bugs%2F2025%2F04%2F03%2Fmsg088095.html&data=05%7C02%7Cbrook%40biology.nmsu.edu%7Cce313e1520f249b728b708dedb04114e%7Ca3ec87a89fb84158ba8ff11bace1ebaa%7C1%7C0%7C639189007694159583%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C60000%7C%7C%7C&sdata=%2Bv5UsD4R9GqmbVitcRxAG%2F81hAsh4KAIIb0ILBfSbhc%3D&reserved=0

OK, thanks.  I have a patch, see below, that solves most of this.  It reverts the commit to src/distrib/utils/embedded/mkimage (rev 1.91) and moves the hashing to all (that I can find) users of mkimage.  This allows for a different use case for evbarm (mkimage -> uncompressed + later compression) versus the others (mkimage -> compressed).  It also solves the problem of only a single hash being created.

One problem remains: all the hashes are appended to MD5/SHA512, which means if those files exist from a prior run of build.sh then more hashes will be added to the old ones.  This is perhaps not quite correct.

Any suggestions on the correct handling of multiple runs is welcome, as is how to clean up gzimg prior to creating files there if that is the thing to do.  Alternatively, suggestions are welcome for retaining only the last hash for any of the files.

One final note: this does not integrate hashes in gzimg with hashes in sets, which is still likely a useful step.

Thanks for your feedback.

Cheers,
Brook


Index: distrib/utils/embedded/mkimage
===================================================================
RCS file: /cvsroot/src/distrib/utils/embedded/mkimage,v
retrieving revision 1.97
diff -u -r1.97 mkimage
--- distrib/utils/embedded/mkimage	22 May 2026 14:44:16 -0000	1.97
+++ distrib/utils/embedded/mkimage	6 Jul 2026 16:18:33 -0000
@@ -425,8 +425,4 @@
 	image="${image}.gz"
 fi
 
-cd "${IMAGEDIR}"
-${CKSUM} -a MD5 "$(basename "${image}")" > MD5
-${CKSUM} -a SHA512 "$(basename "${image}")" > SHA512
-
 echo ${bar} Image is ${image} ${bar}
Index: etc/etc.evbarm/Makefile.inc
===================================================================
RCS file: /cvsroot/src/etc/etc.evbarm/Makefile.inc,v
retrieving revision 1.138
diff -u -r1.138 Makefile.inc
--- etc/etc.evbarm/Makefile.inc	20 Feb 2026 07:20:28 -0000	1.138
+++ etc/etc.evbarm/Makefile.inc	6 Jul 2026 16:18:34 -0000
@@ -96,8 +96,6 @@
 	MACHINE=${MACHINE} \
 	MKDTB=${MKDTB} \
 	MKPOSTFIX=${MKPOSTFIX} \
-	CKSUM=${CKSUM} \
-	IMAGEDIR=${IMAGE.dir} \
 	${HOST_SH} ${MKIMAGE} -h ${.TARGET:S/smp_//} -D ${DESTDIR} \
 	    -S ${NETBSDSRCDIR} -B ${IMAGEENDIAN} ${MKI_OPTS.${.TARGET}} \
 	    ${IMAGE.dir}/${.TARGET:S/smp_//}.img
@@ -107,10 +105,13 @@
 	@${TOOL_INSTALLBOOT} -m ${MACHINE} -o board=${f} \
 			${IMAGE.dir}/${.TARGET:S/smp_//}-${f}.img \
 		&& ${TOOL_GZIP} -f ${IMAGE.dir}/${.TARGET:S/smp_//}-${f}.img \
+		&& (cd ${IMAGE.dir} && (for i in MD5 SHA512; do ${CKSUM} -a $${i} ${.TARGET:S/smp_//}-${f}.img.gz >> $${i}; done)) \
 		|| rm ${IMAGE.dir}/${.TARGET:S/smp_//}-${f}.img
 .endfor
 	@echo "===> Compressing image ${IMAGE.dir}/${.TARGET:S/smp_//}.img"
 	${TOOL_GZIP} -f ${GZIP_N_FLAG} -9 ${IMAGE.dir}/${.TARGET:S/smp_//}.img
+	@echo "===> Calculating hashes for image ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz"
+	(cd ${IMAGE.dir} && (for i in MD5 SHA512; do ${CKSUM} -a $${i} ${.TARGET:S/smp_//}.img.gz >> $${i}; done))
 
 ARCHES.earmv4=		armv4
 ARCHES.earmv4eb=	armv4
Index: etc/etc.evbmips/Makefile.inc
===================================================================
RCS file: /cvsroot/src/etc/etc.evbmips/Makefile.inc,v
retrieving revision 1.32
diff -u -r1.32 Makefile.inc
--- etc/etc.evbmips/Makefile.inc	22 Mar 2025 09:25:14 -0000	1.32
+++ etc/etc.evbmips/Makefile.inc	6 Jul 2026 16:18:34 -0000
@@ -70,11 +70,11 @@
 	MACHINE=${MACHINE} \
 	MKDTB=${MKDTB} \
 	MKPOSTFIX=${MKPOSTFIX} \
-	CKSUM=${CKSUM} \
-	IMAGEDIR=${IMAGE.dir} \
 	${HOST_SH} ${MKIMAGE} -h ${.TARGET:S/smp_//} -D ${DESTDIR} \
             -S ${NETBSDSRCDIR} -B ${IMAGEENDIAN} ${MKI_OPTS.${.TARGET}} \
             ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz
+        @echo "===> Calculating hashes for image ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz"
+	(cd ${IMAGE.dir} && (for i in MD5 SHA512; do ${CKSUM} -a $${i} ${.TARGET:S/smp_//}.img.gz >> $${i}; done))
 
 .if !empty(MACHINE_ARCH:Mmips*64eb)
 smp_octeon: __mkimage
Index: etc/etc.evbppc/Makefile.inc
===================================================================
RCS file: /cvsroot/src/etc/etc.evbppc/Makefile.inc,v
retrieving revision 1.22
diff -u -r1.22 Makefile.inc
--- etc/etc.evbppc/Makefile.inc	10 Jan 2026 06:56:58 -0000	1.22
+++ etc/etc.evbppc/Makefile.inc	6 Jul 2026 16:18:34 -0000
@@ -58,11 +58,11 @@
 	MACHINE=${MACHINE} \
 	MKDTB=${MKDTB} \
 	MKPOSTFIX=${MKPOSTFIX} \
-	CKSUM=${CKSUM} \
-	IMAGEDIR=${IMAGE.dir} \
 	${HOST_SH} -x ${MKIMAGE} -h ${.TARGET:S/smp_//} -D ${DESTDIR} \
             -S ${NETBSDSRCDIR} -B ${IMAGEENDIAN} ${MKI_OPTS.${.TARGET}} \
             ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz
+        @echo "===> Calculating hashes for image ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz"
+        (cd ${IMAGE.dir} && (for i in MD5 SHA512; do ${CKSUM} -a $${i} ${.TARGET:S/smp_//}.img.gz >> $${i}; done))
 
 smp_nintendo: __mkimage
 	cd ${KERNSRCDIR}/arch/evbppc/stand/wii && ${MAKE} release
Index: etc/etc.riscv/Makefile.inc
===================================================================
RCS file: /cvsroot/src/etc/etc.riscv/Makefile.inc,v
retrieving revision 1.7
diff -u -r1.7 Makefile.inc
--- etc/etc.riscv/Makefile.inc	22 Mar 2025 09:25:14 -0000	1.7
+++ etc/etc.riscv/Makefile.inc	6 Jul 2026 16:18:34 -0000
@@ -34,12 +34,11 @@
 	MACHINE=${MACHINE} \
 	MKDTB=${MKDTB} \
 	MKPOSTFIX=${MKPOSTFIX} \
-	CKSUM=${CKSUM} \
-	IMAGEDIR=${IMAGE.dir} \
 	${HOST_SH} ${MKIMAGE} -h ${.TARGET:S/smp_//} -D ${DESTDIR} \
             -S ${NETBSDSRCDIR} ${MKI_OPTS.${.TARGET}} \
             ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz
-
+        @echo "===> Calculating hashes for image ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz"
+        (cd ${IMAGE.dir} && (for i in MD5 SHA512; do ${CKSUM} -a $${i} ${.TARGET:S/smp_//}.img.gz >> $${i}; done))
 
 .if ${MACHINE_ARCH} == "riscv64"
 .if ${HAVE_UEFI} == "yes"