Breakfast: rerun defects first

[epdenouden]

Use case

Rerun defective unit tests first to speed up the red-green-refactor cycle!

Test results and timings are shared between tests runs using a cache. This allows for sorting defective tests to the front of the execution order using an additional sorter. This new sort option is compatible with existing test ordering options.

This project is a followup to my previous work on reordering test execution. The code is available in my Breakfast development branch.

Summary of changes

• the TestSuiteSorter gains the ability to sort suites and tests by priority of defects and execution time
• added a ResultCacheExtension to gather result and timing information during test runs
• added a TestResultCache for sharing of result state between runs
• added caching of run results using CLI flag --cache-result and configuration attribute cacheResult='true'
• added sorting defects to run as quick as possible with CLI flag --order-by=defects and configuration attribute executionOrder='defects'
• ability to specifiy the cache filename using CLI flag --cache-result-file and configuration attribute cacheResultFile
• more extensive testing of sorting scenarios

How to test

1. To activate storing the results of a run use --cache-result:

   phpunit --cache-result
   

2. For this example we switch the cache on for every run by adding the attribute cacheResult="true" to the <phpunit> element in phpunit.xml.

   <phpunit cacheResult="true"> <-- no other changes required --></phpunit>

3. Break TestCaseTest by running it backwards:

   phpunit --order-by=reverse tests/Framework/TestCaseTest.php
   

4. Look at the cache file. By default this is .phpunit.result.cache in the PHPUnit working directory.

5. Break TestCaseTest again with the sorting feature turning on:

   phpunit --order-by=defects tests/Framework/TestCaseTest.php
   

6. The skipped tests are still there but are run immediately now, but still fail. This is easily avoided by asking the sorter to keep dependencies in mind:

   phpunit --order-by=depends,defects tests/Framework/TestCaseTest.php
   

Design considerations

Like the New Order feature the aim is for a robust system with sane defaults that silently does its work and is easily maintained by other developers:

• Require as little new configuration as possible. Just add cacheResult="true" to your configuration and it silently does its work.
• Add no extra output as PHPUnit often lives in automated pipelines and dev scripts.
• Provide robust and maintainable code. This feature also provides a lot of new test coverage for functionality from Add functionality to test dependencies by changing test running order #3092.
• Implemented a simplistic cache on purpose. Obviously people might want more complex caches based on users, groups, configuration, git hash, etc etc. Let's see what people need.