Re: Introducing the patchadd binary patch toolchain

[Adam Hoka <adam.hoka%gmail.com@localhost>]

On Tue, 28 Apr 2009 17:22:02 +0300
Elad Efrat <elad%NetBSD.org@localhost> wrote:

> Hi,
> 
> Tonnerre Lombard wrote:
> > Salut,
> > 
> > Since we're currently releasing NetBSD 5.0, I would like to give a
> > short introduction to the patchadd binary patch toolchain I am
> > currently developing. It is far from finished, but I have a working
> > prototype which can be deployed for the upcoming release already.
> 
> Did you even take a look at haze before you started working on it? when
> you contacted me about your binary patching idea about a year ago I
> pointed you to it. It has most of the code in place you need to add
> binary patch support:
> 
>       http://mail-index.netbsd.org/tech-userlevel/2007/11/06/0001.html
> 
> I also disagree with agc@ about patchadd being the tool that's furthest
> along -- but perhaps he was referring to patching specifically.
> 
> > The tools
> > =========
> > 
> > The tools, which have been added to pkgsrc/sysutils/patchadd recently,
> 
> So you've added an incomplete tool to update the system into mainline
> pkgsrc? out of pure curiosity (and since this is a tool to update
> NetBSD, supposedly), did you discuss it in a public forum as you do
> now?
> 
> > consist of the following tools:
> > 
> >  - patch_add, a tool to apply binary patches,
> >  - patch_delete, a tool to back out a patch added without -r, and
> >  - patch_info, a tool to display the list of installed patches.
> 
> Why are they all separate tools and not a single one?
> 
> > Still missing are the tools:
> > 
> >  - patch_tool, a tool to figure out which patches still need to be
> >    applied, and optionally install them, and
> >  - mkpatchdir, a tool to generate a directory index in a directory full
> >    of patches.
> >  - Potentially even a tool to generate patch sets from two directories
> >    or such.
> > 
> > I'm working on them and I'm consequently going to rewrite the other
> > tools. Until then, use the existing tools. The future tools will be
> > compatible with the current file format. (And syntax.)
> > 
> > The patch format
> > ================
> > 
> > The second important part about patchadd is how to create a binary
> > patch set to be applied. The format basically follows the instructions
> > laid down in http://github.com/tonnerre/patchadd-tests/
> > 
> > The patch file itself is basically a tar file. The file contains:
> > 
> >  - Arbitrarily named bsdiff(1) patches lifting the file from old to new.
> >  - A file called +INFO, containing pure metadata about the patch one
> >    information per line, such as:
> >    * ABI: the OS ABI (e.g. ABI=NetBSD)
> >    * OS_VERSION: the OS version (e.g. OS_VERSION=5.0)
> >    * MACHINE_ARCH: the machine architecture (e.g. MACHINE_ARCH=vax)
> >    * PATCHTOOLS: the file format version (e.g. PATCHTOOLS=0.1)
> >    * NAME: The name of the patch, frequently the file name. A form for
> >      this would have to be agreed on; I would suggest the pullup
> >      request ID (e.g. NAME=4123). Can be free form though, and does not
> >      have to be the same as the file name.
> >    * VERSION: The version of the patch. A patch can be re-released in a
> >      new version if appropriate, and would be suggested as an upgrade
> >      by patch_tool.
> >    * COMMENT: A punchline describing the text. This should be a short
> >      description which is shown in patch_infos overview output.
> >    * DEPENDS: The name of a different patch file to depend on. This
> >      line may appear as many times as required, but please only add one
> >      dependent patch per line.
> >    * CONFLICTS: The name of a different patch which cannot coexist with
> >      this patch. The same rules as for DEPENDS apply.
> >  - A file called +COMMENT containing a longer description (optional).
> >  - A file called +CONTENTS, containing information about each contained
> >    binary diff (one line per file, tab separated):
> >    * The name of the file in the system to be patched, e.g. /bin/ksh
> >    * The name of the patch file in the diff, free form file name
> >    * The SHA-1 sum the file is expected to be in before patching, and
> >    * the SHA-1 sum the file is supposed to have after patching.
> 
> I don't understand why invent yet another file format and syntax. I also
> highly recommend you think about common uses for updating tools (because
> hopefully that's what we want - a *single* tool to be used to update the
> system, and not a confusing collection) -- therefore, you should think
> on how you can make life easier for people who release patches, write
> security advisories/notes based on the patches, and so on.
> 
> Needless to say, haze did all that almost two years ago.
> 
> > The patch index
> > ===============
> > 
> > The patches shall be kept in a directory called
> > ftp://ftp.netbsd.org/${path}/${arch}/${vers}/, e.g. 
> > ftp://ftp.netbsd.org/pub/NetBSD/misc/tonnerre/binpatches/amd64/5.0/
> > 
> > The directory shall contain a file named patchindex. This file starts
> > with a version statement:
> > 
> > Version 0.1
> > 
> > to indicate that the patch file format version 0.1 is used. This
> > statement is followed by a list of all patch metadata for each patch,
> > starting with the Patch keyword and ending with the EndPatch keyword.
> > Multiple consecutive words in a value have to be quoted. For example:
> > 
> > Patch 4123
> > ABI=NetBSD
> > OS_VERSION=4.0
> > [...]
> > COMMENT="A patch to add a funny hat to stargazer"
> > EndPatch
> > 
> > The patch index is then amended with an OpenSSL s/mime inline text
> > signature executed with the patch signing key.
> > 
> > This index file is used solely for the patch_tool utility. Even without
> > this patch index, people can still safely run patch_add 4123 to add the
> > patch.
> 
> Is that another format and syntax?
> 
> > I think this might be a good point to start from.
> 
> Out of personal interest, can you elaborate on why you chose to write a
> tool from scratch, given nbupdate (agc@) and haze (mine) freely
> available? also, can you elaborate on why you prefer binary patching to
> simply distributing replacement files?

Replacement files are much easier to handle IMHO, and they also dont need to 
depend on teach other when you patch the same file several times.

I dont really understand why would we need another patch tool when we have 
Elad's works already in a working state (And by the way, I think it provides 
better functionality than Tonnere's, at least its more innovative).

Also, Elad, its been a long time a had a look at haze, but how do you handle
larger patch bundles?

-- 
Adam Hoka <adam.hoka%gmail.com@localhost>