Re: cvs/fossil time Re: "hg clone https://anonhg.netbsd.org/src/" still aborts, but...

[Reinoud Zandijk <reinoud%NetBSD.org@localhost>]

On Wed, Jun 17, 2020 at 09:14:15AM +0200, Martin Husemann wrote:
> It doesn't take that long for me (I just tried on a fresh dir). I have
> the "evolve" extension installed, which adds like 5m to the final time
> for cache updates (see the lengthy warning below).
> 
> So I'd estimate ~40m for regular installations (on this machine and 
> with my connection to whatever cdn cache is closest, I think decix
> Frankfurt in my case)

I tested the `hg clone' in an empty directory on FFS on my amd64 machine, an
aging Intel i7/920 @ 2.67 GHz quadcore with 12 G mem and a rotating SATA disc
with FFSv2 and WAPBL running stock NetBSD-9.0. Just for refence here are my
findings on this machine and some of my thoughts about it. I used sysstat for
monitoring.

2814.257u 549.599s 1:08:03.85 82.3%     0+0k 14156+79479io 47pf+0w

As you can see, it took quite some time. I had expected the download to go
fine and quick since the speed to the nearest CDN can reach a 10 Mbyte/sec and
expected the hard disc to be the bottleneck but it turned out not to be.

The changeset loading was fast and only took say 5 mins. What really took long
was the manifests stage that took a 30 mins or so and really was completely
hung on the CPU (used one core only of course) and also hardly used the disc
at all. The file changes stage was more puzzling; it also took in the tens of
minutes long but systat didn't show that much CPU and not much hard disc usage
either but from a burst now and then. The phases after that were hung on CPU
usage again with hardly any disc usage.

Only the update stage really starting to hog the disc but that didn't take
that long; a 5+ mins or so; i guess this is just the writing out.

Is there a more reliable way to time each stage with its IO?

What could it be that makes it take that long? Sure parallelism would help, if
possible at all, but it being so hogging the CPU puzzled me; i had expected
the harddisc to be saturated completely but it never was until the last stage.

Reinoud

> time hg clone https://anonhg.NetBSD.org/src
destination directory: src
applying clone bundle from
https://anonhg.NetBSD.org/_bundles/src/77d2a2ece3a06d837da45acd0fda80086ab4113c.zstd.hg
adding changesets
adding manifests                                                                                        
adding file changes                                                                                                         
added 931876 changesets with 2425841 changes to 439702 files (+417 heads)
finished applying clone bundle
searching for changes
adding changesets
adding manifests                                                                                                            
adding file changes                                                                                                         
added 1528 changesets with 9301 changes to 5648 files (+1 heads)                                                            
new changesets 26c8f37631b6:18b9896431a8
931876 local changesets published
updating to bookmark @ on branch trunk
172011 files updated, 0 files merged, 0 files removed, 0 files unresolved                                                   
2814.257u 549.599s 1:08:03.85 82.3%     0+0k 14156+79479io 47pf+0w
>