Re: Update schedule for binary packages?

[Greg Troxel <gdt%lexort.com@localhost>]

"J. Lewis Muir" <jlmuir%imca-cat.org@localhost> writes:

>>   current branch, supported OS version:
>>     i386/amd64, reasonably frequent
>
> What does "reasonably frequent" mean?  I'm wishing for something more
> concrete.

It means that there are a bunch of domU workers for each branch/os
tuple, and they more or less update/build/publish in a loop, except that
probably there isn't enough cpu/ram to run them all at once so sometimes
any one might be paused.

And, I don't know what you are worried about.  Once a build is done,
then yes there are pullups for security, but there aren't a lot of
them.

Really - that's how it is.

If you want things done in a 24h kind of turnaround, then your options
are probably do it yourself or find somebody to pay to do it for you.
Or perhaps donate some build infrastructure to NetBSD (which means a big
computer, remotely manageable, and the costs to host it in an adequately
safe machine room, plus of course there's  the volunteer labor to manage
it).

I think the situation for 8/i386 nad 8/amd64 on the latest stable branch
is pretty good.

> Or I'm wishing for a way to determine that for myself.  For example,
> if there were a web page that listed the status of bulk builds and a
> history of the output/report from those, I could have a look at that
> and presumably see when the last run was, how frequently they run, how
> long they take to run, etc.
>
> I looked at
>
>   https://mail-index.netbsd.org/pkgsrc-bulk/
>
> but that's a really poor UI for finding the information I'm looking for.

I summarized it for you :-)  But the basic problem is that the reality is
fuzzier than you would like...

>> There is no schedule, really.  There are just processes that lead to
>> i386/amd64 getting updated,
>
> It would be nice if there were a schedule.  Even something like the
> following would be helpful: "A bulk build is scheduled to run once a
> day, but sometimes it takes up to two days to complete, in which case it
> will start a day after it last completed."

As I understand it, statements like that are just not true.  The
update/build/post cycle runs as there are resources.  And perhaps
doesn't run again even if there are resources, if the tree has not
changed.

The last commit on the stable branch is

  From: "Benny Siegert" <bsiegert%netbsd.org@localhost>
  Subject: CVS commit: [pkgsrc-2019Q2] pkgsrc/audio/fasttracker2
  To: pkgsrc-changes%NetBSD.org@localhost
  Date: Tue, 23 Jul 2019 13:05:46 +0000 (2 weeks, 2 days, 10 hours ago)
  Reply-To: bsiegert%netbsd.org@localhost

(and thank you bsiegert for doing this work!) and there are builds for
all 4 release/arch combos I'm talking about within 3 days.  So things
really seem pretty good.

>> and the bulk build reports are on the pkgsrc-bulk list.
>
> This is a really poor UI for finding what I'm interested in.  Most of
> the messages are from Joyent Packages Development; nothing against them
> at all, but if I'm looking for a report that corresponds to a particular
> set of binary packages on the NetBSD CDN, it's quite difficult to
> find.

I use gnus to look over the messages that I've stored in an IMAP folder
and find that I can answer these questions pretty easily.
For 8.0/x86_64, I see reports:

E  [ 434: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 8.0/x86_64 2019-06-30 20:05
E  [ 428: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 8.0/x86_64 2019-07-22 22:41
E  [ 446: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 8.0/x86_64 2019-07-25 14:09

for 8.0/i386 I see

EF [ 577: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 8.0/i386 2019-06-30 20:03
E  [ 585: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 8.0/i386 2019-07-15 19:10
E  [ 590: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 8.0/i386 2019-07-18 08:41
E  [ 590: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 8.0/i386 2019-07-20 03:12
E  [ 595: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 8.0/i386 2019-07-22 07:44
E  [ 593: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 8.0/i386 2019-07-24 04:28
E  [ 593: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 8.0/i386 2019-07-26 19:53

NetBSD 7.1 amd64 and i386:

E  [ 453: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 7.1/x86_64 2019-06-30 20:36
E  [ 448: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 7.1/x86_64 2019-07-17 10:24
E  [ 453: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 7.1/x86_64 2019-07-18 14:28
E  [ 442: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 7.1/x86_64 2019-07-23 13:20

E  [ 506: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 7.1/i386 2019-06-30 20:35
E  [ 503: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 7.1/i386 2019-07-12 18:28
E  [ 501: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 7.1/i386 2019-07-14 15:53
E  [ 505: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 7.1/i386 2019-07-18 14:00
E  [ 489: Manuel Bouyer          ] pkgsrc-2019Q2 NetBSD 7.1/i386 2019-07-23 14:13

which basically tells you that those four are pretty up to date.

There are some "bulktracker" pages, like
https://bulktracker.appspot.com/ but I don't think they show what you want.

Agreed this could be better.  I suspect that people have priorities like
having more builds and fixing packages, vs posting stats about build
frequency.

> The other messages appear to be from individual developers,
> and I have no idea if they are reports for the binary packages on
> the NetBSD CDN or for another repository hosted by those individual
> developers.

Some of them are for bulk builds that do not lead to published packages.
Part of the point is to know what failed.

> And still, it's difficult to find the report for the
> os-arch-os_ver-pkgsrc_tag combination I'm looking for.
>
> I'm aware of
>
>   https://cdn.netbsd.org/pub/pkgsrc/packages/reports/
>
> but it doesn't seem to be populated with much at all.  If it were,
> I'd use it.  There are a few reports for earmv{6,7}hf, but that's it;
> nothing for amd64 or any of the other architectures.

The messages give links, usually.  I follow them to try to debug
failures so we can have more packages built.