> Date: Wed, 30 Oct 1996 15:54:40 +0100 (MEZ)
> From: Markus Baeurle <emw4maba@gp.fht-esslingen.de>
> 
> On Tue, 29 Oct 1996, Mark Brinicombe wrote:
> 
>> Yep screw up time again. The rational was that
>> 1. I did not have a spare HD at the time.
>> 2. Time was critical and I really needed to get it out
>> 3. The only change was to the inst script should be no problems.
>> 4. I was doing it in the middle of the night.
>> 
>> Anyway sorry, I should have properly tested it first.
> 
> Normally, the 1.2-beta should be the test for 1.2-release.

I agree.

> It's a pity a new bug was introduced by fixing the ones reported for the 
> install disk, but such is life.

As far as I know, the install disk had major changes between 1.2-beta
and 1.2-release. This was due to problems with 1.2-beta as you say.

> But there were other install problems with 1.2 which were not reported for 
> 1.2-beta, so this wasn't thoroughly tested by anyone I'm afraid.

There were problems with 1.2-release which were not in 1.2-beta.

>> Yep I agree. I would be happy for someone to suggest solutions.
>> At the moment I build the release, test install it, test run it etc.
>> In alot of situations (e.g. programming) the person that creates a program is
>> not the best person to find the bugs. Often there may be things that I take for
>> granted that someone will or will not do and this can cause problems if I have
>> not anticipated everything.
> 
> Well this is not a problem for the beta releases, it's the nature of beta 
> releases.
> The 1.2-beta should have been the test for 1.2-release, but this didn't 
> work for several reasons.
> One was that due to the length of the release process, a lot of 
> improvements or bug fixes were made which lead to other problems.

Yes, that is for sure.

>> Also for me time is a major factor. It is far quicker for me to build a release
>> than to test it properly. To test it I need a spare big disc to do a complete
> 
> Time is THE major factor for you. You're the expert for kernel 
> programming and you should not be buried under a lot of other work.
> But you're a workaholic and load a lot of things on your shoulders which 
> other people could also do.

I couldn't agree more, and that is why I have joined the documentation
project. To take some of the documentation load off Mark.
Also, I want to contribute, and the above is the reason for RDP.

> Ideally, you shouldn't be maintaining the WWW and FTP site. Often, you 
> don't even have the time to do that, let alone fix bugs and problems.

Agreed.

> You really shouldn't have to spend a lot of time on producing RiscBSD CDs.
> I can supply them for Germany and it should not be a problem to export 
> them elsewhere, but payment is obviously still a problem.

I believe that SWIFT may be used for this.
This is an international bank transactions scheme.

If I remember to do it, I will investigate this when I get back
from England.

Please remind me if you don't hear anything from me.

> There are several sets in the release which are not basic stuff, IMHO at 
> least bash, bison, ghostscript, ghostview, jpeg, knews, nedit, nspark, 
> perl, ssh, tcl, tcsh, texinfo, tk, xanim, xlock, xpdf, xpm and xv.

I would say that bash, perl, ssh and tcsh should at least be
tested by the kernel team, as these interact with the kernel
quite closely.

On the other sets I agree.

> There is no reason why you have to compile them, they can be done by mere 
> mortals. ;-) They don't have a direct relation with the RiscBSD releases, 
> so they can be updated out of sync from them by somebody else.
> If you need one of these programs, install the set prepared by somebody 
> else. If there is a problem with it, point it out to the porter.
> This could save you a lot of time.

Yes, I agree.

>> installation and then really I should run with that installation for a week or
>> two. However due to development my machine is not a good test platform for
>> stable releases.
> 
> You're not supposed to do that. Beta versions need not work in the first 
> place. It's more important that the bugs are fixed.

I agree.

> As I said above, 1.2-beta was the test for 1.2-release, but due to 
> necessary changes between them, bugs crept in again or were not fixed at 
> all because nobody reported them or they were forgotten.

The problem here was that the 1.2 release was not released until it
was practically too late to fix the bugs.

> I for one was unsure whether I could use the 1.2-beta install disk or if 
> there were differences to 1.2-release which would lead to an incomplete 
> or different installation, so I stuck to the 1.2-release one although 
> this cost me a lot of nerves.

This is something that should be documented in the installation
manual. And the people who maintain this should always test
the new release before release to make sure that the manual is
correct and that things work.

>> For example for each release I need to test *every* binary that is in the
>> release. How ?
>> What I need are a set of scripts that will exercise every single command.
>> I have come across cases where obscure commands had problems. In fact there are
>> several commands I recently found that can generate SEGV's in the 1.2-beta base
>> set, however I don't expect anybody to have ever used those commands. (A
>> rebuild of the binaries fixed the problem.)
> 
> Aren't you demanding too much? You're trying to produce bug free 
> software, but this does not exist.
> If people find bugs, they must be reported and fixed, often on a NetBSD 
> level.
> There are several hundred known bugs which are unfixed in NetBSD.
> I think it's a bad idea to dig out new bugs which nobody has found yet, 
> then. ;-)

I agree with Mark. He has to test every binary that is in the
release. But, there is no point in him testing them all. He should
have a small set of people testing it for him; me for instance.

> Even more important, there are known bugs in the release anyway (broken 
> console, errors writing to floppy, ibuf overflows to name only the worst) 
> which couldn't be fixed yet, so why bother with even more esoteric ones?

Yes, why bother.

> There are shortcomings like the unbearably slow SCSI drivers which I have 
> had to live with for more than a year now. I don't blame anybody for 
> that, I just want to make clear that these are much much much much more 
> important to fix IMHO than a util which SIGSEGVs if it's called with the 
> wrong arguments.

I don't like that either, and I guess that problem would have been 
fixed if much of the work Mark does now had been delegated to other
people.

> OK if commands from the base set fail during normal use this is an issue, 
> but as there were no changes from beta to release this should not be a 
> problem as they would have been found by people using the beta.
>  
>> Should I perhaps build the release and make it available to a small group for
>> testing  prior to final release ? Who would be perpared to test every release
>>  / upgrade ?
> 
> In general, I would. But there's always the problem of time for me, too.

Same for me, but let me put it this way: Spending some hours installing
it, and then run that installation for a while would not take that
much time.

I have spent many days on RiscBSD that I really should have spent
working. But, I do this because I feel that I can contribute and
get a better system. We need to be a bit more dedicated to this.

We can't just sit on the fence waiting for other people to make
RiscBSD. We have to work together with those in the areas in
which we can contribute.

> But why take additional fuss with testing beta versions which are not 
> guaranteed to work anyway?

Normally a beta version is (or should be) the release, only that
it is distributed outside of the development team for further
testing before release.

> And often people don't want to accept delays due to testing, especially 
> as the testers often won't have time to test new stuff immediately.

We should make a mailing list for beta-testers, and an area on the
ftp server with a user account and password that would only be
distributed to the people on the mailing list.

This will then be a list of people who wants to experiment to the
benefit of all others. Not all people want to be beta-testers, but
those who do should be on this list.

When there have been any changes worth testing, the code should
be made available on this special area first, and if the people
on the list find it to be ok, then it should be released.

This does not need to be large changes. It could e.g. be a new
kernel.

Kjetil B.