Current-Users archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: git copies of cvs modules available

At Mon, 26 Oct 2009 21:48:56 +0100,
'Joerg Sonnenberger' wrote:
> On Mon, Oct 26, 2009 at 04:42:39PM -0400, Marko Schütz wrote:
> > Wouldn't it be nice if we'd convert lots (most) PRs to automated tests
> > which would then automatically be run against each and every
> > commit...?
> Who is going to do the work? 

Maybe somebody who just happens to like working on tests more than
working on, say, documentation...

> There are a number of regression tests in src/tests, they can even
> be run automatically. There is work on a fully automated test setup
> as well. 

I'm aware of those.

> It still needs to be prepared though.

I agree, it needs work. 

> > I have used Aegis ( which explicitly
> > supports this approach and found it very helpful.
> The problem is that the approach doesn't scale. You can't run all
> test cases for every single commit. It is simply not feasible. 

You don't need to run all tests on every commit.

With Aegis for example there are 3 kinds of tests:

- tests against the change-set obviously need to pass on the changed source,
- tests against the baseline accompanying a change-set need to fail on
  the baseline prior to the commit, and
- regression tests run all tests against the change-set

When a change is created, you select which kind you need/want and from
which it will be exempt. Not every change needs to run all tests.

Over time Aegis tracks file-test correlations and can then suggests
tests to run based on the files in a change-set.

> It is also questionable how much problems are actually detected that
> way.

Sure, it depends on the kinds of tests contributors will write.

> So you have good test cases, we like to include them. They just don't
> grow on trees.

It needs more than tests alone, like the support for choosing tests
mentioned above.

Best regards,


Attachment: pgp79LOWGDUYv.pgp
Description: PGP signature

Home | Main Index | Thread Index | Old Index