Re: grafana hits pkgsrc limits again

[Jonathan Perkin <jperkin%mnx.io@localhost>]

* On 2024-07-09 at 09:04 BST, Thomas Klausner wrote:

> For the last grafana update, I had to update the distinfo code to
> handle a huge list of go module files that grafana needs.
>
> I just tried updating grafana to 11.1.0 (see wip/grafana), and now I
> run into a similar limit in the checksum step, from
> mk/checksum/checksum.mk:
>
> ${_CHECKSUM_CMD} ${DISTINFO_FILE:tA} ${_CKSUMFILES}
>
> The ${_CKSUMFILES} variable gets too big for the sh(1) command line
> length limit.

Try these two patches:

  https://github.com/TritonDataCenter/pkgsrc/compare/release/trunk...TritonDataCenter:pkgsrc:feature/parallel-digest/trunk

This implements threaded mode in digest, as well as adding an -F flag 
which changes the behaviour of data coming in to be a list of filenames 
to checksum.

An example of the threaded mode speedup on my macOS laptop (passing the 
data through shasum to verify it's identical):

  $ cd wip/grafana

  $ time bmake checksum | shasum; echo $?
  4345957499803ad801a6e4e9aa1bffaeb968c9ef  -

  real	0m6.696s
  user	0m6.350s
  sys	0m0.468s
  0

  $ time bmake TOOLS_PATH.digest='/path/to/patched/digest -j 16' checksum | shasum; echo $?
  4345957499803ad801a6e4e9aa1bffaeb968c9ef  -

  real	0m2.090s
  user	0m7.526s
  sys	0m0.924s
  0

I don't have any patches for pkgsrc to use -F yet, but it'll look 
something like this:

  $ time bmake -v DISTFILES | tr ' ' '\n' | (cd ../../distfiles; /path/to/digest -j 16 -F md5)
  MD5 (grafana-11.1.0.tar.gz) = 73645b7d5fd376b43957994a4e4722da
  MD5 (grafana-11.1.0.linux-amd64.tar.gz) = 09f56f35fcfb42a4d04407d654561fb3
  [...]
  MD5 (buf.build_gen_go_parca-dev_parca_protocolbuffers_go_@v_v1.33.0-20240414232344-9ca06271cb73.1.zip) = d37f47809ffd0e14424c4ef6f065267a

  real	0m1.350s
  user	0m2.982s
  sys	0m5.855s

That should avoid having to use temporary files (which could be tricky 
as we might not have WRKDIR created), as well as providing a decent 
performance improvement on most machines.

--
Jonathan Perkin   -   mnx.io   -   pkgsrc.smartos.org
Open Source Complete Cloud   www.tritondatacenter.com