Test suite does not respect FIRST principle

[Nek-]

Hello,

As a contributor, the fact that tests don't respect FIRST principle was a pain. This should be fixed one way or another so here is the issue about.

Here is an example of failure:

./vendor/bin/behat features/main/configurable.feature:6
./vendor/bin/behat features/main/content_negotiation.feature:24

Executing these 2 tests make the second fail, while if you run the full test suite, you'll get a success. (this is only 1 example in the middle of hundreds)

Another huge problem is the amount of memory the behat test consumes: as tests are not independents you expect to not have to run the full test suite.. Especially because it takes too much memory to execute.

[soyuka]

#1687

[dunglas]

They are E2E tests, E2E tests are usually not independent (we also have unit tests).
Each feature file is be independent, scenarios however aren't. I'm not fond of #1687 because it will slow down the builds.

[Nek-]

Maybe a memory storage could be a solution. Dropping the schema may be done by removing the db file for performance (in the case of the current storage).

I don't see why E2E tests should not be independent. It's about testing and same rules actually apply. This issue is about contributor pain (which I felt): it is an argument by itself.

[jdeniau]

About the memory consumtion, behat is really greedy.

One fact is also that api-platform disable the cache feature with kernel.debug. It is handy when running a simple test without having to worry about it, but it means that every entity are parsed on every http call, which is awfully long when several tests are running.

For the record on our codebase, we had to set kernel.debug to false in order to make the tests "humanly runable", whith a cache clear functionnality when behat starts.

The code looks like this:

# behat.yml
default:
    extensions:
        Behat\Symfony2Extension:
            kernel:
                debug: false

// FeatureContext.php
class FeatureContext 
{
    private static $needCacheClear = false;

    /**
     * @BeforeSuite
     */
    public static function setClearCache()
    {
        $noClearCache = getenv('BEHAT_NO_CLEAR_CACHE') === 'true';
        self::$needCacheClear =  !$noClearCache;
    }

    /**
     * @BeforeScenario
     */
    public function clearCache()
    {
        if (self::$needCacheClear === true) {
            echo "Clearing cache\n";
            foreach ($this->cacheList as $cache) {
                $cache->clear();
            }
            echo "Cache cleared\n";

            self::$needCacheClear = false;
        }
    }
}

I think it is pretty bad, but I did not found a better way.

If anyone thinks of a better solution, it might be great to speed up api-platform tests and all tests suites that run upon it.

[dunglas]

@Nek- they are independent on a per feature-file basis. Usually, our feature files are small... That being say, I'll of course merge patches improving FIRST principles (it means that it shouldn't degrade the F part).

@jdeniau what we can maybe do if introducing a new parameter like api_platform.metadata_cache, setting it to true in our test suite and defaulting to kernel.debug when not set? Let me try this.

[Simperfit]

The paremeter has been introduced, @Nek- do you want to work on making the tests suite improving this principle ?