Re: Randomize configuration for automatic builds?

["Kamil Rytarowski" <n54%gmx.com@localhost>]

Ian D. Leroux wrote:
> On Mon, May 18, 2015, at 02:27, Kamil Rytarowski wrote:
> > What do you think of introduction in the build machines randomized
> > configuration of mk.conf(5) in each build?
> 
> While it'd certainly be nice if the non-standard options I use were more
> regularly tested, I think the overall value of the automated builds and
> tests is much higher if they are reproducible.  Randomizing the test
> conditions might turn up some bugs that otherwise go unnoticed, but it
> would make it impossible to narrow down which set of changes caused a
> particular failure, because we could never be sure whether the
> difference
> between a successful build and a failed one was in the source code or in
> the build options.

Both types of bugs are fatal, aren't they? Like switching on catman and
disabling info results in failure (and it's actually true).

I see no technical problem in caching used mk.conf. This way we would
penetrate all possible configs with just a matter of time including the
default one.

> 
> Now if we had enough resources to test non-standard configs
> *systematically*, that might be interesting, but in my experience
> breakage related to my non-standard config is so much rarer than
> breakage/fixage in the standard builds that I doubt it'd be worth
> the effort.
> 

OK, anyway I will try to do it occasionally and locally.

The catches problems are rather cosmetic, but still halt the proper build.