Re: creating unique urls for binary packages

[Jonathan Perkin <jperkin%joyent.com@localhost>]

* On 2017-05-18 at 07:53 BST, Jonathan Perkin wrote:

> * On 2017-05-18 at 06:24 BST, matthew sporleder wrote:
> 
> > A few days ago we ran into a situation where some packages were
> > rebuilt but not given new names, causing pkgin and friends to be upset
> > about checksums of files not matching pkg_summary because
> > cdn.netbsd.org had the packages cached from the previous build.
> > 
> > There are a few ways to resolve this, but the easiest one is to have
> > the client (pkgin) append a unique identifier (like the build date) to
> > the package's url:
> > bash-2.05.2.13.tgz?BUILD_DATE=2017-01-07_23:23:31_+0000
> > 
> > when it fetched that package.
> > 
> > Thoughts?
> 
> This seems like a bug in the CDN?  pkgin itself should handle rebuilt
> packages ok, as long as you are running a version with this commit:
> 
>   https://github.com/NetBSDfr/pkgin/commit/f21d2ec9
> 
> Are you not enabling ETags or something?

I forgot to mention, one of the main issues around providing a binary
package repository is ensuring that updates to the repo are atomic, or
at least as tight as possible, as all of the packages need to be in
sync with pkg_summary and its checksums of them all.

To help fix that in our environment (we rebuild more than most so have
run into this quite a bit in the past) I have this pbulk patch:

  https://github.com/joyent/pkgsrc/commit/7c32dd9

and I have a similar mechanism in the script which is used to mirror
packages across our pkgsrc.joyent.com mirrors.

Since making those changes and the pkgin one above we've had much
fewer complaints about bad checksums - I don't remember the last time
we had one.

-- 
Jonathan Perkin  -  Joyent, Inc.  -  www.joyent.com