toolchain/57463: issues with build to DESTDIR on case insenstive file system

[lukem%netbsd.org@localhost]

>Number:         57463
>Category:       toolchain
>Synopsis:       issues with build to DESTDIR on case insenstive file system
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    toolchain-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Jun 10 19:55:00 +0000 2023
>Originator:     Luke Mewburn
>Release:        NetBSD-current as at 2023-07-10
>Organization:
NetBSD
>Environment:
>Description:

tl;dr: building on case insenstive file systems will result in
sets will missing files and/or files with incorrect contents
because of duplicate case-insensitive file names, mostly
manual pages.


Various files installed into DESTDIR that have conflicting names
if DESTDIR is on a case-insensitve file system won't
be available; only one of the duplicates will exist.

This isn't obvious at first, since during "make install" portion of the
build the conflicting-if-case-insensitve filenames just get overwritten
without an error, and the sets appear to be generated fine.

I first observed this when running on a macOS host
	% ./build.sh -D DESTDIR -E install=ROOTDIR
where these are both on case insensitive file systems,
which failed with:
	pax: Cannot link file ROOTDIR/./usr/share/man/html9/FREE.html to itself
	pax: Cannot link file ROOTDIR/./usr/share/man/html9/ISSET.html to itself


Checking the METALOG for case-insensitive matches of "9/free" using
	% grep -i 9/free DESTDIR/METALOG

On a NetBSD host:
	./usr/share/man/html9/FREE.html type=file uname=root gname=wheel mode=0444 size=12225 sha256=871393d86f3f8fa1f94fe44458e1daf2cfbb06bacdbc974a882c073a25d06558
	./usr/share/man/html9/free.html type=file uname=root gname=wheel mode=0444 size=12225 sha256=871393d86f3f8fa1f94fe44458e1daf2cfbb06bacdbc974a882c073a25d06558
	./usr/share/man/man9/FREE.9 type=file uname=root gname=wheel mode=0444 size=6806 sha256=f23c599362b8a57aaf568ded6e9633f86eefcd5b057daa239c1b949d9c74e171
	./usr/share/man/man9/free.9 type=file uname=root gname=wheel mode=0444 size=6806 sha256=f23c599362b8a57aaf568ded6e9633f86eefcd5b057daa239c1b949d9c74e171

On a case-insenstive macOS host:
	./usr/share/man/html9/FREE.html type=file uname=root gname=wheel mode=0444 size=12225 sha256=871393d86f3f8fa1f94fe44458e1daf2cfbb06bacdbc974a882c073a25d06558
	./usr/share/man/html9/free.html type=file uname=root gname=wheel mode=0444 size=12225 sha256=871393d86f3f8fa1f94fe44458e1daf2cfbb06bacdbc974a882c073a25d06558
	./usr/share/man/man9/free.9 type=file uname=root gname=wheel mode=0444 size=6806 sha256=f23c599362b8a57aaf568ded6e9633f86eefcd5b057daa239c1b949d9c74e171

Notice usr/share/man/man9/FREE.9 is missing?


Checking the output of the comp.tar.xz set built on case insensitive
macOS host versus a NetBSD host showed that some of the files were
stored differently too.

On a NetBSD host:
	% tar -Jtvf SETDIR/comp.tar.xz | grep -i 9/free | wc -l
	18

On a macOS host:
	% tar -Jtvf SETDIR/comp.tar.xz | grep -i 9/free | wc -l
	8


A method to determine all the potential case-insensitive filename
conflicts is to build to a DESTDIR on a NetBSD host and run:
	% cd DESTDIR
	% find . -type f | tr A-Z a-z | sort | uniq -d

E.g.,:
	./usr/share/man/html2/_exit.html
	./usr/share/man/html3/gcq_init.html
	./usr/share/man/html3/gcq_init_head.html
	./usr/share/man/html4/dbcool.html
	./usr/share/man/html4/i386/pcibios.html
	./usr/share/man/html8/makedev.html
	./usr/share/man/html8/makedev.local.html
	./usr/share/man/html9/cardbus_conf_read.html
	./usr/share/man/html9/cardbus_conf_write.html
	./usr/share/man/html9/cardbus_free_tag.html
	./usr/share/man/html9/cardbus_function_disable.html
	./usr/share/man/html9/cardbus_function_enable.html
	./usr/share/man/html9/cardbus_make_tag.html
	./usr/share/man/html9/cardbus_mapreg_map.html
	./usr/share/man/html9/cardbus_mapreg_unmap.html
	./usr/share/man/html9/delay.html
	./usr/share/man/html9/free.html
	./usr/share/man/html9/isset.html
	./usr/share/man/html9/knote.html
	./usr/share/man/html9/m_prepend.html
	./usr/share/man/html9/malloc.html
	./usr/share/man/man2/_exit.2
	./usr/share/man/man3/gcq_init.3
	./usr/share/man/man3/gcq_init_head.3
	./usr/share/man/man4/dbcool.4
	./usr/share/man/man4/i386/pcibios.4
	./usr/share/man/man8/makedev.8
	./usr/share/man/man8/makedev.local.8
	./usr/share/man/man9/cardbus_conf_read.9
	./usr/share/man/man9/cardbus_conf_write.9
	./usr/share/man/man9/cardbus_free_tag.9
	./usr/share/man/man9/cardbus_function_disable.9
	./usr/share/man/man9/cardbus_function_enable.9
	./usr/share/man/man9/cardbus_make_tag.9
	./usr/share/man/man9/cardbus_mapreg_map.9
	./usr/share/man/man9/cardbus_mapreg_unmap.9
	./usr/share/man/man9/delay.9
	./usr/share/man/man9/free.9
	./usr/share/man/man9/isset.9
	./usr/share/man/man9/knote.9
	./usr/share/man/man9/m_prepend.9
	./usr/share/man/man9/malloc.9

>How-To-Repeat:
Build releases and sets on a case insenstive file system.
Carefuly compare the differences in the generated sets.

>Fix:
To be determined.

It's long been a goal of build.sh to support building on systems
with case insensitive file systems, and our CVS source was adapted
many years ago to allow source checkouts on that.