[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: creating unique urls for binary packages
> On May 18, 2017, at 3:12 AM, Jonathan Perkin <jperkin%joyent.com@localhost> wrote:
> * On 2017-05-18 at 07:53 BST, Jonathan Perkin wrote:
>> * On 2017-05-18 at 06:24 BST, matthew sporleder wrote:
>>> A few days ago we ran into a situation where some packages were
>>> rebuilt but not given new names, causing pkgin and friends to be upset
>>> about checksums of files not matching pkg_summary because
>>> cdn.netbsd.org had the packages cached from the previous build.
>>> There are a few ways to resolve this, but the easiest one is to have
>>> the client (pkgin) append a unique identifier (like the build date) to
>>> the package's url:
>>> when it fetched that package.
>> This seems like a bug in the CDN? pkgin itself should handle rebuilt
>> packages ok, as long as you are running a version with this commit:
>> Are you not enabling ETags or something?
A hit is a hit is a hit. We use cache control max-age not must-revalidate or similar.
Revalidating each time is an option but is slower than just allowing a pure edge hit. Also since netbsd has been trying to dogfood bozo I am not super confident if-modified-since is going to do anything. I don't think bozohttp has etags. Switching to apache is not a big deal to solve some of these if it comes to it.
The query param idea is just one option. We can also use a header + Vary or use a smart purging strategy on build, but a URL param is just super easy.
> I forgot to mention, one of the main issues around providing a binary
> package repository is ensuring that updates to the repo are atomic, or
> at least as tight as possible, as all of the packages need to be in
> sync with pkg_summary and its checksums of them all.
> To help fix that in our environment (we rebuild more than most so have
> run into this quite a bit in the past) I have this pbulk patch:
Good idea. I am not that involved with the bulk build origin part though.
> and I have a similar mechanism in the script which is used to mirror
> packages across our pkgsrc.joyent.com mirrors.
> Since making those changes and the pkgin one above we've had much
> fewer complaints about bad checksums - I don't remember the last time
> we had one.
Are you using must-revalidate or something else?
Main Index |
Thread Index |