Re: Fix mk/fetch/github.mk

[Greg Troxel <gdt%lexort.com@localhost>]

Thomas Klausner <wiz%gatalith.at@localhost> writes:

> On Tue, Feb 13, 2024 at 06:39:03PM -0500, Greg Troxel wrote:
>> Overall, my impressions are (and they are a little fuzzy):
>> 
>>   GITHUB_TAG should only be used for things that are actual git tags
>> 
>>   GITHUB_HASH (bikeshed, making that up) should be used for things that
>>   are (currently SHA-1) commit hashes
>> 
>>   I am unclear on if there is anything else.
>
> Github tags and releases are different things.

Sorry, I left out releases by accident while knowing about them.  I
agree that releases are different, and think they are what projects
should be creating.  I think they are fetched differently from tags and
commit hashes.

> Tags are the result of 'git tag' and a push of the tag to github.  The
> downloaded tarball for that is a snapshot of the git repository.

Agreed.

> On the github web frontend you can make a release (based on a tag),
> which allows you to add release notes and also upload separate
> tarballs (which might include e.g. autoconf-generated files) - and
> these tarballs are the files that are fetched by the pkgsrc
> infrastructure.

I think there is a release tarball if you don't try to do anything
special that is just like a tag tarball.  But, not our problem at least
today.

> The names of the tags and releases can be arbitrarily set, but there
> are conventions - usually "${PKGREVSION}" or "v${PKGVERSION}" which is
> why your list looks the way it looks.

Agreed.

> The download URL logic (on the github side, I think, not sure about
> that) automatically expands some values. And in very rare cases, that
> is not unique, which is what adam's commit tries to address.

Certainly this needs to get better.  It is just seeming tricky to figure
out exactly how.

Are we having collisions between sha-1 hashes and tags?  If not, what is
colliding?  I have so far not absorbed the original problem.

>>   1187 v${PKGVERSION_NOREV}
>>    159 ${PKGVERSION_NOREV}
>>     57 ${DISTNAME}
>>     34 v${VERSION}
>>     20 refs/tags/v${PKGVERSION_NOREV}
>
> This is the usual expansion when you use v${PKGVERSION_NOREV} and what
> url2pkg provides - I don't think anyone adds these manually.

Do you mean nobody adds refs/tags manually?

I have a fuzzy memory of having to add refs/tags to GITHUB_TAG for
something to get it to download a tag tarball.  Or maybe I was just
making it match the URL that I grabbed from my browser from a download
link.

Do we think that v${PKGVERSION_NOREV} and refs/tags/v${PKGVERSION_NOREV}
are usually equivalent?

>>     13 release-${PKGVERSION_NOREV}
>
> As I mentioned, conventions only... (more like that skipped).
>
> So yes, there's at least three things - tags, releases, and git commit hashes.
>
> Hope this helps,

It helps a lot, because you didn't say there were more beyond the
three.

What are the canonical download URLs for tags and commit hashes?

With the recent change, unison-snapshot tries to fetch:

  https://github.com/bcpierce00/unison/archive/refs/tags/d9bb5734b802b23e1ca3553fdeb8e2d15d45e0ac.tar.gz

which fails.  With up-to-date pkgsrc, it succeeds in fetching

  Requesting https://github.com/bcpierce00/unison/archive/d9bb5734b802b23e1ca3553fdeb8e2d15d45e0ac.tar.gz
  Redirected to https://codeload.github.com/bcpierce00/unison/tar.gz/d9bb5734b802b23e1ca3553fdeb8e2d15d45e0ac

With lensfun, I see

  Requesting https://github.com/lensfun/lensfun/archive/v0.3.4.tar.gz
  Redirected to https://codeload.github.com/lensfun/lensfun/tar.gz/refs/tags/v0.3.4
  Requesting https://codeload.github.com/lensfun/lensfun/tar.gz/refs/tags/v0.3.4

So it seems there is "tar.gz" for commit hashes and "refs/tags" for
tags.

I am suggesting to encode GITHUB_TAG vs GITHUB_HASH in packages, so that
we can adapt the fetch logic over time to fetch those kinds of things,
which are logically separate.

Right now we have a mix of objects in TAG.  A path could be:

  add GITHUB_HASH (and document of course)

  enhance url2pkg to choose HASH/TAG correctly

  flip packages with commit hashes to GITHUB_HASH

  change fetch url for GITHUB_TAG

which should result in relatively little breakage.