Wednesday 8 July 2009

On no! Not more tests!

One of the problems with the incredible level of success of the PHP TestFest is that PHP gets more tests, 887 more to be precise.

Well, isn't that the point? Yes, but it's beginning to take *forever* to run them which is a bad thing and as TestFests get bigger and better this will only get worse.

A year or so ago I started worrying about this, then I met Stefan Priebsch at IPC and made him worry about it too. Together we started a project to re-factor the code that runs tests (run-tests.php) so that it could run tests in parallel.

In May we had a prototype going, it ran most of the tests in sequence. It still doesn't have all of the capability of run-tests.php but it was enough for our PHP GSOC student (Georg Gradwohl) to start writing and integrating code that would run groups of tests in parallel.

Yesterday I ran some tests on my dual core lap top - this isn't a proper benchmark and I had all sorts of other stuff going on too...but here they are:



The chart shows the time taken to run 7700 tests with the current code (old), new code sequentially (seq) and the new code with increasing numbers of parallel tasks (para). As you can see we are able to run this sample in about half the time taken by the code that is used at the moment. There is still a long way to go with this but I'm happy to say that a couple of people have started to check out the code and try and run it. At the moment this only works on *ix, supporting Windows is on the TODO list.

With the help of the 'many hands' of the PHP community I'm quietly confident that by the time PHP gets all the tests it needs we will have something that will run them efficiently :-)

No comments: