At Mon, 26 Oct 2009 21:48:56 +0100, 'Joerg Sonnenberger' wrote: > > On Mon, Oct 26, 2009 at 04:42:39PM -0400, Marko Schütz wrote: > > Wouldn't it be nice if we'd convert lots (most) PRs to automated tests > > which would then automatically be run against each and every > > commit...? > > Who is going to do the work? Maybe somebody who just happens to like working on tests more than working on, say, documentation... > There are a number of regression tests in src/tests, they can even > be run automatically. There is work on a fully automated test setup > as well. I'm aware of those. > It still needs to be prepared though. I agree, it needs work. > > I have used Aegis (http://aegis.sourceforge.net/) which explicitly > > supports this approach and found it very helpful. > > The problem is that the approach doesn't scale. You can't run all > test cases for every single commit. It is simply not feasible. You don't need to run all tests on every commit. With Aegis for example there are 3 kinds of tests: - tests against the change-set obviously need to pass on the changed source, - tests against the baseline accompanying a change-set need to fail on the baseline prior to the commit, and - regression tests run all tests against the change-set When a change is created, you select which kind you need/want and from which it will be exempt. Not every change needs to run all tests. Over time Aegis tracks file-test correlations and can then suggests tests to run based on the files in a change-set. > It is also questionable how much problems are actually detected that > way. Sure, it depends on the kinds of tests contributors will write. > So you have good test cases, we like to include them. They just don't > grow on trees. It needs more than tests alone, like the support for choosing tests mentioned above. Best regards, Marko
Attachment:
pgp79LOWGDUYv.pgp
Description: PGP signature