Re: gzimg hashes

["David H. Gutteridge" <david%gutteridge.ca@localhost>]

On Tue, 2026-07-07 at 10:52 -0600, Brook Milligan wrote:
> 
> > On Jul 6, 2026, at 10:49, Brook Milligan <brook%biology.nmsu.edu@localhost> wrote:
> > 
> > 
> > > On Jul 5, 2026, at 20:12, David H. Gutteridge <david%gutteridge.ca@localhost> wrote:
> > > 
> > > On Sun, 5 Jul 2026 at 17:42:17 -0600, Brook Milligan wrote:
> > > > I notice that the hashes generated for release files in the gzimg
> > > > directory cover the uncompressed files, not the compressed files
> > > > actually distributed.  This seems to be a mistake.
> > > > 
> > > > The fix is clearly to reorder the hash generation relative to the
> > > > compression, but I’m checking here to make sure I’m not missing
> > > > something.
> > > > 
> > > > Cheers,
> > > > Brook
> > > 
> > > Yes, that's one of the issues raised in PR 59195 (and its duplicate,
> > > PR 60302), e.g.,
> > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail-index.netbsd.org%2Fnetbsd-bugs%2F2025%2F04%2F03%2Fmsg088095.html&data=05%7C02%7Cbrook%40biology.nmsu.edu%7Cce313e1520f249b728b708dedb04114e%7Ca3ec87a89fb84158ba8ff11bace1ebaa%7C1%7C0%7C639189007694159583%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C60000%7C%7C%7C&sdata=%2Bv5UsD4R9GqmbVitcRxAG%2F81hAsh4KAIIb0ILBfSbhc%3D&reserved=0
> > 
> > OK, thanks.  I have a patch, see below, that solves most of this.  
> 
> Here is (I believe) a better patch + the referenced mkhashes script.  It does the following:
> 
> - reverts the commit to src/distrib/utils/embedded/mkimage (rev 1.91) so mkimage is no longer responsible for hashes
> 
> - creates the script src/distrib/utils/embedded/mkimage, which makes a standard set of hashes.  The boilerplate is based upon mkimage, so I hope that is OK.
> 
> - uses mkhashes to generate hashes for all (that I know of) users, i.e., in src/etc/etc.*/Makefile.inc
> 
> Note that only a single hash for any file will be retained and multiple hashes will accumulate in the files.
> 
> I think this solves the issues mentioned in PR 59195 and PR 60302.
> 
> Comments are welcome.
> 
> Thanks a lot.
> 
> Cheers,
> Brook

Hello,

As is, it doesn't work correctly for me with builds where there's more
than one resulting image, e.g.:

[disciple@arcusxx:gzimg]$ cat MD5                                                                                                                                           
MD5 (arm64mbr.img) = 8db3453cda03be499e0c663ccb608cf1
MD5 (arm64.img.gz) = 688ce2c196c9aa86ddb5ca4577f0942f
[disciple@arcusxx:gzimg]$ cksum -cw MD5
(MD5) arm64mbr.img: FAILED
[disciple@arcusxx:gzimg]$ md5 arm64.img.gz                                                                                                                                  
MD5 (arm64.img.gz) = 688ce2c196c9aa86ddb5ca4577f0942f
[disciple@arcusxx:gzimg]$ md5 arm64mbr.img.gz                                                                                                                               
MD5 (arm64mbr.img.gz) = 4d3ae51f9a103ab6e08a1f4b81b77f19
[disciple@arcusxx:gzimg]$ cat SHA512                                                                                                                                        
SHA512 (arm64mbr.img) = d788ac9d38ff026f97d2504f57f19a719d966c6abdf3cd8aec70eea69a2546ff3b758ac4596ada8a58eeeabb112a825e8e8c1dc5946cfeb450d2fa91398bc2b4
SHA512 (arm64.img.gz) = bc0c519eaf4f207bcff34842a67bab67b44bc2d387234c07ef79d2f4562bd4f7702eedabf92308aafd347063ebb9a4ec13daed4c020847d494460595c71a322a
[disciple@arcusxx:gzimg]$ cksum -cw SHA512                                                                                                                                  
(SHA512) arm64mbr.img: FAILED
[disciple@arcusxx:gzimg]$ sha512 arm64.img.gz                                                                                                                               
SHA512 (arm64.img.gz) = bc0c519eaf4f207bcff34842a67bab67b44bc2d387234c07ef79d2f4562bd4f7702eedabf92308aafd347063ebb9a4ec13daed4c020847d494460595c71a322a
[disciple@arcusxx:gzimg]$ sha512 arm64mbr.img.gz                                                                                                                            
SHA512 (arm64mbr.img.gz) = 7543ee99b9e8af9e6d86fc1beb39bbbbaaf62a7352dd6bb70d9d50849c92d0ba4cd21d29b393f2854b7b3fb872dbeaf536fcf6f9170dd112ea1046daa01fdfc3

This is where I've built the "release" target from a clean state, then
done an update build afterward (once further changes to the tree
occurred).

(I had a different approach I imagined for this, but I'm not sure it
would work as I expect (I haven't tried), and would also be ugly in
that I think it would require a cleanup make target in a different file
than where the rest of the action occurs.)

A couple of other comments follow in-line below.

> 
> Index: distrib/utils/embedded/mkimage
> ===================================================================
> RCS file: /cvsroot/src/distrib/utils/embedded/mkimage,v
> retrieving revision 1.97
> diff -u -r1.97 mkimage
> --- distrib/utils/embedded/mkimage	22 May 2026 14:44:16 -0000	1.97
> +++ distrib/utils/embedded/mkimage	7 Jul 2026 16:36:35 -0000
> @@ -425,8 +425,4 @@
>  	image="${image}.gz"
>  fi
>  
> -cd "${IMAGEDIR}"
> -${CKSUM} -a MD5 "$(basename "${image}")" > MD5
> -${CKSUM} -a SHA512 "$(basename "${image}")" > SHA512
> -
>  echo ${bar} Image is ${image} ${bar}
> Index: etc/etc.evbarm/Makefile.inc
> ===================================================================
> RCS file: /cvsroot/src/etc/etc.evbarm/Makefile.inc,v
> retrieving revision 1.138
> diff -u -r1.138 Makefile.inc
> --- etc/etc.evbarm/Makefile.inc	20 Feb 2026 07:20:28 -0000	1.138
> +++ etc/etc.evbarm/Makefile.inc	7 Jul 2026 16:36:35 -0000
> @@ -3,6 +3,7 @@
>  #	etc.evbarm/Makefile.inc -- evbarm-specific etc Makefile targets
>  #
>  
> +MKHASHES=${NETBSDSRCDIR}/distrib/utils/embedded/mkhashes
>  MKIMAGE= ${NETBSDSRCDIR}/distrib/utils/embedded/mkimage
>  
>  # If you change the list of distributed kernels, don't forget
> @@ -96,8 +97,6 @@
>  	MACHINE=${MACHINE} \
>  	MKDTB=${MKDTB} \
>  	MKPOSTFIX=${MKPOSTFIX} \
> -	CKSUM=${CKSUM} \
> -	IMAGEDIR=${IMAGE.dir} \
>  	${HOST_SH} ${MKIMAGE} -h ${.TARGET:S/smp_//} -D ${DESTDIR} \
>  	    -S ${NETBSDSRCDIR} -B ${IMAGEENDIAN} ${MKI_OPTS.${.TARGET}} \
>  	    ${IMAGE.dir}/${.TARGET:S/smp_//}.img
> @@ -107,10 +106,12 @@
>  	@${TOOL_INSTALLBOOT} -m ${MACHINE} -o board=${f} \
>  			${IMAGE.dir}/${.TARGET:S/smp_//}-${f}.img \
>  		&& ${TOOL_GZIP} -f ${IMAGE.dir}/${.TARGET:S/smp_//}-${f}.img \
> +		&& ${MKHASHES} -c ${CKSUM} ${IMAGE.dir}/${.TARGET:S/smp_//}-${f}.img.gz \
>  		|| rm ${IMAGE.dir}/${.TARGET:S/smp_//}-${f}.img
>  .endfor
>  	@echo "===> Compressing image ${IMAGE.dir}/${.TARGET:S/smp_//}.img"
>  	${TOOL_GZIP} -f ${GZIP_N_FLAG} -9 ${IMAGE.dir}/${.TARGET:S/smp_//}.img
> +	${MKHASHES} -c ${CKSUM} ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz 
>  
>  ARCHES.earmv4=		armv4
>  ARCHES.earmv4eb=	armv4
> Index: etc/etc.evbmips/Makefile.inc
> ===================================================================
> RCS file: /cvsroot/src/etc/etc.evbmips/Makefile.inc,v
> retrieving revision 1.32
> diff -u -r1.32 Makefile.inc
> --- etc/etc.evbmips/Makefile.inc	22 Mar 2025 09:25:14 -0000	1.32
> +++ etc/etc.evbmips/Makefile.inc	7 Jul 2026 16:36:35 -0000
> @@ -52,6 +52,9 @@
>  IMAGEENDIAN=	le
>  .endif
>  
> +# Support for mkhashes
> +MKHASHES= 	${NETBSDSRCDIR}/distrib/utils/embedded/mkhashes
> +
>  __mkimage: .USE
>  	TOOL_MAKE=${MAKE} \
>  	TOOL_MAKEFS=${TOOL_MAKEFS} \
> @@ -70,11 +73,10 @@
>  	MACHINE=${MACHINE} \
>  	MKDTB=${MKDTB} \
>  	MKPOSTFIX=${MKPOSTFIX} \
> -	CKSUM=${CKSUM} \
> -	IMAGEDIR=${IMAGE.dir} \
>  	${HOST_SH} ${MKIMAGE} -h ${.TARGET:S/smp_//} -D ${DESTDIR} \
>              -S ${NETBSDSRCDIR} -B ${IMAGEENDIAN} ${MKI_OPTS.${.TARGET}} \
>              ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz
> +	${MKHASHES} -c ${CKSUM} ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz
>  
>  .if !empty(MACHINE_ARCH:Mmips*64eb)
>  smp_octeon: __mkimage
> Index: etc/etc.evbppc/Makefile.inc
> ===================================================================
> RCS file: /cvsroot/src/etc/etc.evbppc/Makefile.inc,v
> retrieving revision 1.22
> diff -u -r1.22 Makefile.inc
> --- etc/etc.evbppc/Makefile.inc	10 Jan 2026 06:56:58 -0000	1.22
> +++ etc/etc.evbppc/Makefile.inc	7 Jul 2026 16:36:35 -0000
> @@ -39,6 +39,9 @@
>  IMAGE.kern=     ${IMAGE.rel}/binary/kernel
>  IMAGEENDIAN=	be
>  
> +# Support for mkhashes
> +MKHASHES= 	${NETBSDSRCDIR}/distrib/utils/embedded/mkhashes
> +
>  .if ${MACHINE_ARCH} == "powerpc"
>  __mkimage: .USE
>  	TOOL_MAKE=${MAKE} \
> @@ -58,11 +61,10 @@
>  	MACHINE=${MACHINE} \
>  	MKDTB=${MKDTB} \
>  	MKPOSTFIX=${MKPOSTFIX} \
> -	CKSUM=${CKSUM} \
> -	IMAGEDIR=${IMAGE.dir} \
>  	${HOST_SH} -x ${MKIMAGE} -h ${.TARGET:S/smp_//} -D ${DESTDIR} \
>              -S ${NETBSDSRCDIR} -B ${IMAGEENDIAN} ${MKI_OPTS.${.TARGET}} \
>              ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz
> +	${MKHASHES} -c ${CKSUM} ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz
>  
>  smp_nintendo: __mkimage
>  	cd ${KERNSRCDIR}/arch/evbppc/stand/wii && ${MAKE} release
> Index: etc/etc.riscv/Makefile.inc
> ===================================================================
> RCS file: /cvsroot/src/etc/etc.riscv/Makefile.inc,v
> retrieving revision 1.7
> diff -u -r1.7 Makefile.inc
> --- etc/etc.riscv/Makefile.inc	22 Mar 2025 09:25:14 -0000	1.7
> +++ etc/etc.riscv/Makefile.inc	7 Jul 2026 16:36:35 -0000
> @@ -15,6 +15,9 @@
>  IMAGE.dir=      ${IMAGE.rel}/binary/gzimg
>  IMAGE.kern=     ${IMAGE.rel}/binary/kernel
>  
> +# Support for mkhashes
> +MKHASHES= 	${NETBSDSRCDIR}/distrib/utils/embedded/mkhashes
> +
>  #	TOOL_MKUBOOTIMAGE=${TOOL_MKUBOOTIMAGE} \
>  
>  __mkimage: .USE
> @@ -34,12 +37,10 @@
>  	MACHINE=${MACHINE} \
>  	MKDTB=${MKDTB} \
>  	MKPOSTFIX=${MKPOSTFIX} \
> -	CKSUM=${CKSUM} \
> -	IMAGEDIR=${IMAGE.dir} \
>  	${HOST_SH} ${MKIMAGE} -h ${.TARGET:S/smp_//} -D ${DESTDIR} \
>              -S ${NETBSDSRCDIR} ${MKI_OPTS.${.TARGET}} \
>              ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz
> -
> +	${MKHASHES} -c ${CKSUM} ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz
>  
>  .if ${MACHINE_ARCH} == "riscv64"
>  .if ${HAVE_UEFI} == "yes"
> 
> 
> #!/bin/sh
> # $NetBSD$
> #
> # Copyright (c) 2026 The NetBSD Foundation, Inc.
> # All rights reserved.
> #
> # This code is derived from software contributed to The NetBSD Foundation
> # by Brook Milligan.
> #
> # Redistribution and use in source and binary forms, with or without
> # modification, are permitted provided that the following conditions
> # are met:
> # 1. Redistributions of source code must retain the above copyright
> #    notice, this list of conditions and the following disclaimer.
> # 2. Redistributions in binary form must reproduce the above copyright
> #    notice, this list of conditions and the following disclaimer in the
> #    documentation and/or other materials provided with the distribution.
> # 3. Neither the name of The NetBSD Foundation nor the names of its
> #    contributors may be used to endorse or promote products derived
> #    from this software without specific prior written permission.
> #
> # THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS
> # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
> # TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
> # PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
> # BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
> # CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> # SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
> # INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
> # CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
> # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
> # POSSIBILITY OF SUCH DAMAGE.
> #

The licensing terms standard for NetBSD is the two clause version,
rather than three: https://www.netbsd.org/about/redistribution.html
(I see mkimage has the older version; no one has changed it.)

> 
> #
> # Makes a set of hashes for a file.
> #
> 
> set -e
> 
> PROG="$(basename "$0")"
> 
> cksum=${CKSUM:-cksum}
> hashes=${HASHES:-"MD5 SHA512"}
> imagedir=${IMAGEDIR}
> 
> bar="==="
> 
> tmp="$(mktemp -d "${TMPDIR:-/tmp}/$PROG.XXXXXX")"

mktemp(1) is not part of POSIX (AFAIK) and is not guaranteed to be
present on a given build host (even though that should be highly likely
in our context), so this should be using the host tools framework, not
a bare "mktemp" reference. (As is already done with cksum(1) here.)
(The same is probably true for grep(1), since it's also part of the
host tools.)

> 
> trap "cleanup" 0 1 2 3 15
> 
> cleanup() {
> 	case "$tmp" in
> 	"${TMPDIR:-/tmp}/$PROG."*)	rm -fr "$tmp";;
> 	esac
> }
> 
> usage() {
> 	cat << EOF 1>&2
> Usage: $PROG [-H <hashes>] [-c <cksum>] [-d <imagedir>] <image>
> 
> -H	Hash algorithms [${hashes}]
> -c	Checksum program [${cksum}]
> -d	Image directory [dirname \${image}]
> EOF
> 	exit 1
> }
> 
> OPTS="H:c:d:"
> while getopts "$OPTS" f
> do
> 	case $f in
> 	H)	hashes="$OPTARG";;
> 	c)	cksum="$OPTARG";;
> 	d)	imagedir="$OPTARG";;
> 	*)	usage;;
> 	esac
> done
> 
> shift $(( $OPTIND - 1 ))
> if [ -n "$1" ]; then
> 	# take the next argument as being the image name
> 	image="$1"
> 	shift
> fi
> 
> if [ -z "${image}" ]; then
> 	echo "${PROG}: image undefined"
> 	usage
> fi
> 
> if [ -z "${imagedir}" ]; then
> 	imagedir="$(dirname "${image}")"
> fi
> 
> image="$(basename "${image}")"
> 
> cd "${imagedir}"
> 
> if [ ! -f ${image} ]; then
> 	echo "${PROG}: ${imagedir}/${image} does not exist"
> 	usage
> fi
> 
> for h in ${hashes}; do
> 	echo "${bar} ${h} hash: ${imagedir}/${image} ${bar}"
> 	if [ -f ${h} ]; then
> 		grep -v "${image}" ${h} > ${tmp}/${h} || true
> 	else
> 		cp /dev/null ${tmp}/${h}
> 	fi
> 	${cksum} -a ${h} "${image}" >> ${tmp}/${h}
> 	mv ${tmp}/${h} ${h}
> done

Sorry for the delayed response, my sometimes meagre time for NetBSD was
mostly taken up by a recent hardware failure. I could run more tests
and look at what's happening in more detail soon.

Thanks,

Dave