On Tue, 29 Oct 1996, Mark Brinicombe wrote:

> Yep screw up time again. The rational was that
> 1. I did not have a spare HD at the time.
> 2. Time was critical and I really needed to get it out
> 3. The only change was to the inst script should be no problems.
> 4. I was doing it in the middle of the night.
> 
> Anyway sorry, I should have properly tested it first.

Normally, the 1.2-beta should be the test for 1.2-release.
It's a pity a new bug was introduced by fixing the ones reported for the 
install disk, but such is life.
But there were other install problems with 1.2 which were not reported for 
1.2-beta, so this wasn't thoroughly tested by anyone I'm afraid.

Also an upgrade disk for 1.2-beta was missing, otherwise I would have 
tested it. I thought there wasn't one any more so I didn't ask why it was 
missing.
 
> Yep I agree. I would be happy for someone to suggest solutions.
> At the moment I build the release, test install it, test run it etc.
> In alot of situations (e.g. programming) the person that creates a program is
> not the best person to find the bugs. Often there may be things that I take for
> granted that someone will or will not do and this can cause problems if I have
> not anticipated everything.

Well this is not a problem for the beta releases, it's the nature of beta 
releases.
The 1.2-beta should have been the test for 1.2-release, but this didn't 
work for several reasons.
One was that due to the length of the release process, a lot of 
improvements or bug fixes were made which lead to other problems.

> Also for me time is a major factor. It is far quicker for me to build a release
> than to test it properly. To test it I need a spare big disc to do a complete

Time is THE major factor for you. You're the expert for kernel 
programming and you should not be buried under a lot of other work.
But you're a workaholic and load a lot of things on your shoulders which 
other people could also do.

Ideally, you shouldn't be maintaining the WWW and FTP site. Often, you 
don't even have the time to do that, let alone fix bugs and problems.

You really shouldn't have to spend a lot of time on producing RiscBSD CDs.
I can supply them for Germany and it should not be a problem to export 
them elsewhere, but payment is obviously still a problem.

There are several sets in the release which are not basic stuff, IMHO at 
least bash, bison, ghostscript, ghostview, jpeg, knews, nedit, nspark, 
perl, ssh, tcl, tcsh, texinfo, tk, xanim, xlock, xpdf, xpm and xv.
There is no reason why you have to compile them, they can be done by mere 
mortals. ;-) They don't have a direct relation with the RiscBSD releases, 
so they can be updated out of sync from them by somebody else.
If you need one of these programs, install the set prepared by somebody 
else. If there is a problem with it, point it out to the porter.
This could save you a lot of time.

> installation and then really I should run with that installation for a week or
> two. However due to development my machine is not a good test platform for
> stable releases.

You're not supposed to do that. Beta versions need not work in the first 
place. It's more important that the bugs are fixed.
As I said above, 1.2-beta was the test for 1.2-release, but due to 
necessary changes between them, bugs crept in again or were not fixed at 
all because nobody reported them or they were forgotten.
I for one was unsure whether I could use the 1.2-beta install disk or if 
there were differences to 1.2-release which would lead to an incomplete 
or different installation, so I stuck to the 1.2-release one although 
this cost me a lot of nerves.
 
> For example for each release I need to test *every* binary that is in the
> release. How ?
> What I need are a set of scripts that will exercise every single command.
> I have come across cases where obscure commands had problems. In fact there are
> several commands I recently found that can generate SEGV's in the 1.2-beta base
> set, however I don't expect anybody to have ever used those commands. (A
> rebuild of the binaries fixed the problem.)

Aren't you demanding too much? You're trying to produce bug free 
software, but this does not exist.
If people find bugs, they must be reported and fixed, often on a NetBSD 
level.
There are several hundred known bugs which are unfixed in NetBSD.
I think it's a bad idea to dig out new bugs which nobody has found yet, 
then. ;-)

Even more important, there are known bugs in the release anyway (broken 
console, errors writing to floppy, ibuf overflows to name only the worst) 
which couldn't be fixed yet, so why bother with even more esoteric ones?

There are shortcomings like the unbearably slow SCSI drivers which I have 
had to live with for more than a year now. I don't blame anybody for 
that, I just want to make clear that these are much much much much more 
important to fix IMHO than a util which SIGSEGVs if it's called with the 
wrong arguments.

OK if commands from the base set fail during normal use this is an issue, 
but as there were no changes from beta to release this should not be a 
problem as they would have been found by people using the beta.
 
> Should I perhaps build the release and make it available to a small group for
> testing  prior to final release ? Who would be perpared to test every release
>  / upgrade ?

In general, I would. But there's always the problem of time for me, too.

But why take additional fuss with testing beta versions which are not 
guaranteed to work anyway?
And often people don't want to accept delays due to testing, especially 
as the testers often won't have time to test new stuff immediately.

So long, Markus Baeurle