Re: toolchain/47562 (very slow installworld)

[lukem%NetBSD.org@localhost]

Synopsis: very slow installworld

Responsible-Changed-From-To: toolchain-manager->lukem
Responsible-Changed-By: lukem%NetBSD.org@localhost
Responsible-Changed-When: Sat, 10 Jun 2023 17:50:09 +0000
Responsible-Changed-Why:


State-Changed-From-To: open->analyzed
State-Changed-By: lukem%NetBSD.org@localhost
State-Changed-When: Sat, 10 Jun 2023 17:50:09 +0000
State-Changed-Why:

I've looked into the history of the mtree/sets installation from before
2013-02-13 (per your "Recent changes" in the description).  I couldn't
see anything specifically obvious.

There were distrib/sets changes around 2013-02-06 by Christos
to use mtree to generate the set lists when the METALOG isn't
present (when MKUNPRIVED=no):
        revision 1.77
        date: 2013-02-06 12:01:12 +1100;  author: christos;  state: Exp;  lines: +8 -8;
        Create full sets using mtree from the flists (not just filenames).

which may be the cause of your "{*LONG PAUSE HERE*}", beacuse
the etc/mtree/sets.* files with their sha256 digests are being
calculated in "installsets" on the (slow) target machine.


In -current I've added more MAKEVERBOSE support into
distrib/sets/Makefile and distrib/sets/maketars;
if you run with MAKEVERBOSE=2 you may get more diagnostic
information to confirm which operations are causing the delays.


I'm not keen on adding an option in distrib/sets to disable
the generation of sha256 in MKUNPRIVED=no installsets, because
then the generation of /etc/mtree/* would then differ from
the MKUNPRIVED=yes equivalent


There's a couple of suggestions to avoid the slow issue:

Rather than explicitly strip out the sha256 in the MKUNPRIVED=no
installsets generation of /etc/mtree/* (which would then
differ from the MKUNPRIVED=yes equivalent), there's a couple
of suggestions to avoid the issue:

1. Cross-build sets on a faster machine and extract those.
   (I assume that's how you built the distribution on nfs?)

2. Cross-build with MKUNPRIVED=yes on a faster machine which
   generates the sha256 as each file is installed.
   Install on the destination with MKUNPRIVED=yes to enable
   using the existing METALOG.

Thoughts?
                                                                              44,4          Bot