Re: creating unique urls for binary packages

[Matt Sporleder <msporleder%gmail.com@localhost>]

> On May 18, 2017, at 3:12 AM, Jonathan Perkin <jperkin%joyent.com@localhost> wrote:
> 
> * On 2017-05-18 at 07:53 BST, Jonathan Perkin wrote:
> 
>> * On 2017-05-18 at 06:24 BST, matthew sporleder wrote:
>> 
>>> A few days ago we ran into a situation where some packages were
>>> rebuilt but not given new names, causing pkgin and friends to be upset
>>> about checksums of files not matching pkg_summary because
>>> cdn.netbsd.org had the packages cached from the previous build.
>>> 
>>> There are a few ways to resolve this, but the easiest one is to have
>>> the client (pkgin) append a unique identifier (like the build date) to
>>> the package's url:
>>> bash-2.05.2.13.tgz?BUILD_DATE=2017-01-07_23:23:31_+0000
>>> 
>>> when it fetched that package.
>>> 
>>> Thoughts?
>> 
>> This seems like a bug in the CDN?  pkgin itself should handle rebuilt
>> packages ok, as long as you are running a version with this commit:
>> 
>>  https://github.com/NetBSDfr/pkgin/commit/f21d2ec9
>> 
>> Are you not enabling ETags or something?
> 

A hit is a hit is a hit. We use cache control max-age not must-revalidate or similar. 

Revalidating each time is an option but is slower than just allowing a pure edge hit. Also since netbsd has been trying to dogfood bozo I am not super confident if-modified-since is going to do anything. I don't think bozohttp has etags.  Switching to apache is not a big deal to solve some of these if it comes to it. 


The query param idea is just one option. We can also use a header + Vary or use a smart purging strategy on build, but a URL param is just super easy. 

> I forgot to mention, one of the main issues around providing a binary
> package repository is ensuring that updates to the repo are atomic, or
> at least as tight as possible, as all of the packages need to be in
> sync with pkg_summary and its checksums of them all.
> 
> To help fix that in our environment (we rebuild more than most so have
> run into this quite a bit in the past) I have this pbulk patch:
> 
>  https://github.com/joyent/pkgsrc/commit/7c32dd9

Good idea. I am not that involved with the bulk build origin part though. 

> 
> and I have a similar mechanism in the script which is used to mirror
> packages across our pkgsrc.joyent.com mirrors.
> 
> Since making those changes and the pkgin one above we've had much
> fewer complaints about bad checksums - I don't remember the last time
> we had one.

Are you using must-revalidate or something else?