APITestCase: Validate OpenApi definition using a json schema

[fherbin]

Description
would it be possible to validate the openApi definition?
I will explain for example it will be easier:
when I want to integrate my swagger definition in my api gateway (gravitee.io), it finds me errors like in the exemple below.

it would be great to be able to get these errors directly in apiplatform, in order to consolidate our openApi definitions.

Example

**attribute paths.'/v1/edit_facture_eaus/{id}' is not of type `object`
attribute paths.'/v1/chorus_pro/factures/upload'(post).[['unknown']].name is missing
attribute paths.'/v1/chorus_pro/factures/upload'(post).responses is missing
attribute paths.'/v1/chorus_pro/factures/{refDette}/status'(get).[refDette].type is missing
attribute paths.'/v1/chorus_pro/factures/{refDette}/status'(get).responses is missing
attribute paths.'/v1/soldes/abonnement/{abonnementId}' Declared path parameter abonnementId needs to be defined as a path parameter in path or operation level
attribute paths.'/v1/soldes/client/{clientId}'. Declared path parameter clientId needs to be defined as a path parameter in path or operation level

[soyuka]

We're using npx swagger-cli validate build/out/openapi/swagger_v2.json, I suggest that you use the same for now:

https://github.com/api-platform/core/blob/master/.github/workflows/ci.yml#L293

Note that you could easily add a schema validation to your test suite using:

• https://raw.githubusercontent.com/OAI/OpenAPI-Specification/master/schemas/v2.0/schema.json (v2 swagger)
• https://github.com/OAI/OpenAPI-Specification/blob/master/schemas/v3.0/schema.json (v3 openapi)

with $this->assertMatchesJsonSchema(/* a JSON Schema as an array or as a string */);

available with our Test Client: https://api-platform.com/docs/core/testing/#the-test-httpclient

I'll add a documentation entry and this will be even easier to test on the next version of API Platform (#3407)

[fherbin]

great thx !

[fherbin]

@soyuka I tried to use the assertion 'assertMatchesJsonSchema' like this:

<?php

namespace App\Tests\OpenApi;

use ApiPlatform\Core\Bridge\Symfony\Bundle\Test\ApiTestCase;

class OpenApiTest extends ApiTestCase
{
    const OPEN_API_JSON_SCHEMA_V2_URL = 'https://raw.githubusercontent.com/OAI/OpenAPI-Specification/master/schemas/v2.0/schema.json';
    const OPEN_API_JSON_SCHEMA_V3_URL = 'https://raw.githubusercontent.com/OAI/OpenAPI-Specification/master/schemas/v3.0/schema.json';

    public function testOpenApiV2Validation()
    {
        static::createClient()->request(
            'GET',
            '/api/docs.json',
            [
                'headers' => ['Accept' => 'application/json'],
                'query' => ['spec_version' => 2],
            ]
        );
        $this->assertMatchesJsonSchema(file_get_contents(self::OPEN_API_JSON_SCHEMA_V2_URL));
    }

    public function testOpenApiV3Validation()
    {
        static::createClient()->request(
            'GET',
            '/api/docs.json',
            [
                'headers' => ['Accept' => 'application/json'],
                'query' => ['spec_version' => 3],
            ]
        );
        $this->assertMatchesJsonSchema(file_get_contents(self::OPEN_API_JSON_SCHEMA_V3_URL));
    }
}

but both don't work, here is the return of phpunit :

www-data@0b46516f9e2e:~/html/msfacturations$  ./bin/phpunit
PHPUnit 7.5.20 by Sebastian Bergmann and contributors.

Testing Project Test Suite
EE                                                                  2 / 2 (100%)

Time: 4.32 seconds, Memory: 46.50 MB

There were 2 errors:

1) App\Tests\OpenApi\OpenApiTest::testOpenApiV2Validation
JsonSchema\Exception\InvalidSchemaMediaTypeException: Media type application/schema+json expected

/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Uri/UriRetriever.php:92
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Uri/UriRetriever.php:209
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Uri/UriRetriever.php:181
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/SchemaStorage.php:52
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/SchemaStorage.php:115
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/SchemaStorage.php:138
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/SchemaStorage.php:162
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Constraints/Constraint.php:118
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Constraints/ObjectConstraint.php:145
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Constraints/ObjectConstraint.php:47
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Constraints/Constraint.php:85
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Constraints/UndefinedConstraint.php:80
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Constraints/UndefinedConstraint.php:51
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Constraints/Constraint.php:118
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Constraints/SchemaConstraint.php:92
/var/www/html/msfacturations/vendor/justinrainbow/json-schema/src/JsonSchema/Validator.php:63
/var/www/html/msfacturations/vendor/api-platform/core/src/Bridge/Symfony/Bundle/Test/Constraint/MatchesJsonSchema.php:63
/var/www/html/msfacturations/vendor/api-platform/core/src/Bridge/Symfony/Bundle/Test/ApiTestAssertionsTrait.php:109
/var/www/html/msfacturations/tests/OpenApi/OpenApiTest.php:22

2) App\Tests\OpenApi\OpenApiTest::testOpenApiV3Validation
Error: Call to a member function export() on null

/var/www/html/msfacturations/vendor/api-platform/core/src/Bridge/Symfony/Bundle/Test/ApiTestAssertionsTrait.php:109
/var/www/html/msfacturations/tests/OpenApi/OpenApiTest.php:35

ERRORS!
Tests: 2, Assertions: 2, Errors: 2.

Is there anything I didn't understand?

[soyuka]

I need to dig a bit further, this should've worked indeed.

[soyuka]

Hi, I tracked the bugs down, your first issue is related to: https://stackoverflow.com/questions/28302900/mediatype-should-be-application-schemajson-to-validate-schema

Basically the Swagger v2 schema uses http://swagger.io/v2/schema.json which doesn't answers with a json content-type, maybe because it's a redirection (301) and because justinrainbow/json-schema uses FileGetContents. I have a fix in mind we need to create our own UriRetrieverInterface that handles this.

The Call to a member function export() on null is way trickier still looking.

[soyuka]

Okay I've found a fix!

    public function testOpenApiV3Validation()
    {
        static::createClient()->request(
            'GET',
            '/api/docs.json',
            [
                'headers' => ['Accept' => 'application/json'],
                'query' => ['spec_version' => 3],
            ]
        );
        $this->assertMatchesJsonSchema(file_get_contents(self::OPEN_API_JSON_SCHEMA_V3_URL), Constraint::CHECK_MODE_TYPE_CAST);
    }

CHECK_MODE_TYPE_CAST looks required because of the ArrayObject transformation. I'm adding a fix for the UriRetriever in the morning.

Thanks @fherbin for testing this!

[fherbin]

ok, works for v3 with this commit applied :
6fce2ad

[fherbin]

Hi, I tracked the bugs down, your first issue is related to: https://stackoverflow.com/questions/28302900/mediatype-should-be-application-schemajson-to-validate-schema

Basically the Swagger v2 schema uses http://swagger.io/v2/schema.json which doesn't answers with a json content-type, maybe because it's a redirection (301) and because justinrainbow/json-schema uses FileGetContents. I have a fix in mind we need to create our own UriRetrieverInterface that handles this.

The Call to a member function export() on null is way trickier still looking.

there is a property for that yes :
https://github.com/justinrainbow/json-schema/blob/fa4d2d3c1e40a222ded125b679891086c9a6c7d6/src/JsonSchema/Uri/UriRetriever.php#L38

[soyuka]

See jsonrainbow/json-schema#646 for a redirection fix.

I've proposed a patch there.

I don't really have any other solution to offer as we don't want to add too much code in API Platform to cover this use case. A solution on your end would be to change the MatchesJsonSchema (copy our implementation) to something like this:

final class MyMatchesJsonSchema extends Constraint
{
    // ...

    protected function matches($other): bool
    {
        $other = $this->normalizeJson($other);

        $validator = $this->getValidator();
        $validator->validate($other, $this->schema, $this->checkMode);

        return $validator->isValid();
    }

    private function getValidator(): Validator
    {
        $uriRetriever = new UriRetriever();
        // Create your own JsonSchema\Uri\Retrievers\UriRetrieverInterface that works properly
        $uriRetriever->setUriRetriever(// insert it here);
        return new Validator(new Factory(null, $uriRetriever));
    }
}

And create your assertion like this:

    /**
     * @param object|array|string $jsonSchema
     */
    public static function assertMatchesJsonSchema($jsonSchema, ?int $checkMode = null, string $message = ''): void
    {
        $constraint = new MyMatchesJsonSchema($jsonSchema, $checkMode);

        static::assertThat(self::getHttpResponse()->toArray(false), $constraint, $message);
    }

[soyuka]

@fherbin about the The Call to a member function export() on null issue, would you be able to update the phpunit version you're using?

[fherbin]

@fherbin about the The Call to a member function export() on null issue, would you be able to update the phpunit version you're using?

PHPUnit 7.5.20 so far, yes it's possible

[...]
2) App\Tests\OpenApiTest::testOpenApiV3Validation
Error: Call to a member function export() on null

/var/www/html/msfacturations/vendor/api-platform/core/src/Bridge/Symfony/Bundle/Test/ApiTestAssertionsTrait.php:109
/var/www/html/msfacturations/tests/openApiTest.php:35

ERRORS!
Tests: 2, Assertions: 2, Errors: 2.
www-data@0b46516f9e2e:~/html/msfacturations$ bin/phpunit --version
PHPUnit 7.5.20 by Sebastian Bergmann and contributors.