Validate your PHP API tests against OpenAPI definitions – a Laravel example

OpenAPI HttpFoundation Testing

OpenAPI is a specification for describing RESTful APIs in JSON and YAML format, aiming at being understandable by both humans and machines.

OpenAPI definitions are language-agnostic and can be used in several ways:

An OpenAPI definition can be used by documentation generation tools to display the API, code generation tools to generate servers and clients in various programming languages, testing tools, and many other use cases.

– The OpenAPI Specification

This article demonstrates how to write integration tests that compare API responses to OpenAPI 3.0.x definitions in order to validate that the former conform to the latter.

We will use the OpenAPI HttpFoundation Testing package in a fresh Laravel installation for which we'll also generate a Swagger UI documentation using the L5 Swagger package.

The first part of the post will describe the issue and provide further context – if you're just here for the code, you're welcome to skip ahead and go to the Laravel example section straight away.

The issue

APIs are commonplace nowadays and the decent ones come with a documentation describing the endpoints and how to use them. These documents come in various shapes and forms, but one thing they've got in common in that they need to be updated every time the API changes.

To many developers, API documentation is an afterthought – it's boring, sometimes tedious, often unrewarding. Using annotations to keep the code and the documentation in one place can help, but these are often painful to write and even the most willing developer is likely to introduce errors that won't necessarily be caught by reviewers.

In this rather common scenario, documentation and API can end up out of sync, leading to frustrated consumers.

Another goal of API maintenance is to ensure that existing endpoints keep functioning properly – regressions can and will appear over time, and may go unnoticed if there isn't a proper testing strategy in place.

One approach is to write integration tests controlling the API's behaviour and automatically detecting breaking changes as soon as they are introduced. While this is a good strategy, it isn't foolproof – there's no guarantee that the expectations set in the tests will always 100% match the documentation.

What we need is a way to ensure they remain in sync.

A solution

Let's assume that we have an API documented with OpenAPI and some integration tests, and that we now want to align the expectations of the latter with the operations listed in the former.

While using the OpenAPI specification is a strong starting point, it's not enough to address the issue straight away.

What sets it apart, however, is the growing number of tools built on top of it, pushing its utility far beyond the documenting aspect.

One tool destined for the PHP community and maintained by The PHP League is OpenAPI PSR-7 Message Validator, a package validating PSR-7 HTTP messages against OpenAPI definitions.

The idea is to take HTTP requests and responses and to compare them with OpenAPI definitions, and return an error if they don't match.

In other words, we can use this package to add an extra layer on top of our integration tests, comparing API responses with the OpenAPI definitions of our API, and making the test fail in case of discrepancy.

Here is a fancy diagram to illustrate the relationship between the API, the integration tests and the OpenAPI definition:

Relationship between OpenAPI, API and tests

The definition describes the API, and the tests use the definition to make sure the API behaves the way the definition says it does.

All of a sudden, our OpenAPI definition becomes a reference for both our code and our tests – it acts as the API's single source of truth.

PSR-7

One caveat of the OpenAPI PSR-7 Message Validator package is that, as its name indicates, it only works with PSR-7 messages.

The issue is that not all frameworks support this standard by default – as a matter of fact, a lot of them use Symfony's HttpFoundation component under the hood, whose requests and responses do not implement that standard out of the box.

The Symfony folks thought of this, however, and provided a bridge that converts HttpFoundation objects to PSR-7 ones. The bridge simply needs a PSR-7 and PSR-17 factory, for which they suggest to use Tobias Nyholm's PSR-7 implementation.

All of these pieces form a jigsaw puzzle that the OpenAPI HttpFoundation Testing package sets out to solve for us, allowing developers to back their integration tests with OpenAPI definitions in applications relying on the HttpFoundation component.

Let's see how to use it in a Laravel project, which falls into this category.

A Laravel example

The code featured in this section is also available as a GitHub repository for reference.

First, let's create a new Laravel 9 project, using Composer:

$ composer create-project --prefer-dist laravel/laravel openapi-example "9.*"

Enter the project's root folder and instal a couple of dependencies:

$ cd openapi-example
$ composer require --dev osteel/openapi-httpfoundation-testing
$ composer require darkaonline/l5-swagger -W

The first one is the OpenAPI HttpFoundation Testing package mentioned earlier, that we instal as a development dependency as it's intended to be used as part of our test suite.

The second one is L5 Swagger, a popular package bringing Swagger PHP and Swagger UI to Laravel. We actually don't need Swagger PHP here, as it uses Doctrine annotations to generate OpenAPI definitions and we're going to manually write our own instead. We do need Swagger UI, however, and the package conveniently adapts it to work with Laravel (the -W option is simply here to also update related dependencies, to avoid conflicts).

To make sure Swagger PHP doesn't overwrite the OpenAPI definition, let's set the following environment variable in the .env file, at the root of the project:

L5_SWAGGER_GENERATE_ALWAYS=false

Create a file named api-docs.yaml in the storage/api-docs folder (which you need to create), and add the following content to it:

openapi: 3.0.3

info:
    title: OpenAPI HttpFoundation Testing Laravel Example
    version: 1.0.0

servers:
    - url: http://localhost:8000/api

paths:
    '/test':
    get:
        responses:
        '200':
            description: Ok
            content:
            application/json:
                schema:
                type: object
                required:
                    - foo
                properties:
                    foo:
                    type: string
                    example: bar

This is a simple OpenAPI definition describing a single operation – a GET request to the /api/test endpoint, that should return a JSON object containing a required foo key.

Let's check whether Swagger UI displays our OpenAPI definition correctly. Start the PHP development server (from the project's root):

$ php artisan serve

Open localhost:8000/api/documentation in your browser and replace api-docs.json with api-docs.yaml in the navigation bar at the top (this is so Swagger UI loads up the YAML definition instead of the JSON one, as we haven't provided the latter).

Hit the enter key or click Explore – our OpenAPI definition should now be rendered as a Swagger UI documentation:

Swagger UI

Expand the /test endpoint and try it out – it should fail with a 404 Not Found error, because we haven't implemented it yet.

Let's fix that now. Open the routes/api.php file and replace the example route with this one:

Route::get('/test', function (Request $request) {
    return response()->json(['foo' => 'bar']);
});

Go back to the Swagger UI tab and try the endpoint again – it should now return a successful response.

Time to write a test! Open tests/Feature/ExampleTest.php and replace its content with this one:

<?php

namespace Tests\Feature;

use Osteel\OpenApi\Testing\ValidatorBuilder;
use Tests\TestCase;

class ExampleTest extends TestCase
{
    /**
     * A basic test example.
     *
     * @return void
     */
    public function testBasicTest()
    {
        $response = $this->get('/api/test');

        $validator = ValidatorBuilder::fromYaml(storage_path('api-docs/api-docs.yaml'))->getValidator();

        $result = $validator->validate($response->baseResponse, '/test', 'get');

        $this->assertTrue($result);
    }
}

Let's unpack this a bit. For those unfamiliar with Laravel, $this->get() is a test method provided by the MakesHttpRequests trait to perform a GET request to the provided endpoint, executing the request's lifecycle without leaving the application. It returns a response that is identical to one we would obtain if we'd perform the same request from the outside.

We then create a validator using the Osteel\OpenApi\Testing\ValidatorBuilder class, to which we feed the YAML definition we wrote earlier via the fromYaml static method (the storage_path function is a helper returning the path to the storage folder, where we placed the definition).

Had we had a JSON definition instead, we could have used the fromJson method. Also, both methods accept YAML and JSON strings respectively, in addition to files.

The builder returns an instance of Osteel\OpenApi\Testing\Validator, on which we call the get method, passing the path and the response as parameters ($response is an instance of Illuminate\Testing\TestResponse here – a wrapper for the underlying HttpFoundation object, which can be retrieved through the baseResponse public property).

The above is basically the equivalent of saying:

I want to validate that this response conforms to the OpenAPI definition of a GET request to the /test path.

It could also be written this way:

$result = $validator->get($response->baseResponse, '/test');

That's because the validator exposes a shortcut method for each of the HTTP methods supported by OpenAPI (GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS and TRACE), to make it simpler to test responses for the corresponding operations.

Note that the specified path must exactly match one of the OpenAPI definition's paths.

You can now run the test, which should be successful:

$ ./vendor/bin/phpunit tests/Feature

Open routes/api.php again, and change the route for this one:

Route::get('/test', function (Request $request) {
    return response()->json(['baz' => 'bar']);
});

Run the test again – it should now fail, because the response contains baz instead of foo, and the OpenAPI definition says the latter is expected.

Our test is officially backed by OpenAPI!

The above is obviously an oversimplified example for the sake of demonstration, but in a real situation a good practice would be to overwrite the MakesHttpRequests trait's call method, so it performs both the test request and the OpenAPI validation.

This way, our test would now fit on one line:

$this->get('/api/test');

This could be implemented as a new MakesOpenApiRequests trait that would "extend" the MakesHttpRequests one, and that would first call the parent call method to get the response. It would then work out the path from the URI, and validate the response against the OpenAPI definition before returning it, for the calling test to perform any further assertions, as needed.

Conclusion

While this setup is a great step towards improving an API's robustness, it is no silver bullet – it requires that every single endpoint is covered with integration tests, which is not easily enforceable in an automated way and ultimately still requires some discipline and vigilance on the part of developers.

It may even feel a bit coercive at first, since as a result maintainers are basically forced to keep the documentation up to date to write successful tests.

The added value, however, is that said documentation is now guaranteed to be much more accurate, leading to happy consumers which, in turn, should lead to less frustrated developers, who will spend less time hunting down pesky discrepancies.

All in all, making OpenAPI definitions the single source of truth for both the API documentation and integration tests is in itself a strong incentive to keep them up to date. They naturally become a priority, where they used to be an afterthought.

As for maintaining the OpenAPI definition itself, doing so manually can feel a bit daunting. Annotations are a solution, but I personally don't like them and prefer to maintain a YAML file directly. IDE extensions like this one for VSCode make it much easier, but if you can't bear the sight of a YAML or JSON file, you can also use tools like Stoplight Studio to do it through a more user-friendly interface.

And since we're talking about Stoplight, this article about API Design-First vs Code First by Phil Sturgeon is a good starting point for API documentation in general, and might help you choose an approach to documenting that works for you.