How can we tell if we're making a difference?

It's quite hard to tell how well the PHP TestFest is succeeding in the aim of improving the PHP test coverage. Next time we do this we should take a snapshot of the coverage before we start - hindsight is a wonderful thing.

Here is what I did to try and find out:

From the CVS change log for May 2008, 101 PHP test files were added or changed in PHP 5.3 under the ~ext directory.

Looking at these I can identify 49 that I know come from TestFest activity, these are 30 new mcrypt tests – we have David Soria Parra to thank for these – having a one-man TestFest there :-)

Of the others there are 17 new DOM tests – for which we thank the London PHP group and 2 new reflection tests that came from the Dutch PHP TestFest. Given that there currently about 5000 tests in PHP, if we only count the tests committed that's about a 1% increase.

So where are the rest of the tests? There are still 73 tests in testfest.php.net waiting to be reviewed and committed – we really need someone who understands reflection to review and commit reflection tests. I notice that there is a new cURL test today as well – that's brave.

What about coverage? I believe that the mcrypt coverage leapt from less than 30% up to over 75% thanks to David's efforts! The DOM coverage will go up by about 3.5% by the time all the London tests are in and there is plenty more to do in that extension. As for reflection - I can't tell but I'd love to know. In case anyone wants to check later it's at 75% today without the the new tests.

In case anyone is looking for places to help, check this for some interesting analysis of what is and is not tested. Oh yes, there's plenty more to do :-)