oasdiff

Given:

• Two versions of the same application: A and B.
• Both versions expose a contract described in OpenAPI.
• The application has consumers that rely on the current contract.

The task:

• Determine whether the new version’s contract introduces any backwards-incompatible changes.
• Determine what additional changes the contract has undergone.

Why not just run diff and look at the differences?

• The spec is large enough that you can’t make sense of the differences “by eye” in a single pass.
• Spotting backwards-incompatible changes by eye is very hard.
• The formatting of identical elements changes depending on the generator version — in one version an array might be one line, in another it spans several, or the elements may be reordered. That inflates the diff and doesn’t make the task any simpler.

The last point isn’t all that hard to solve with jq --sort-keys. A simple example:

diff \
    <(jo foo=1 bar=2 baz=3 | jq --sort-keys) \
    <(jo baz=4 foo=1 bar=2 | jq --sort-keys)
3c3
<   "baz": 3,
---
>   "baz": 4,

But it doesn’t make the main problem any faster or any easier.

What we need is to find or build a tool that shows the differences between two versions of the same spec in a human-readable format.

Fortunately, such solutions exist. Let’s take a quick look at the tools that made it into the survey.

redskap/swagger-brake ¹

• java;
• solid documentation;
• couldn’t chew through the schemas we fed it.

civisanalytics/swagger-diff ²

• ruby;
• good README;
• trips on the schemas we passed in and bails out with an error.

Sayi/swagger-diff ³

• java;
• to build a working jar from source you’d have to patch pom.xml;
• couldn’t digest the schemas we fed it.

atlassian/openapi-diff ⁴

• ts;
• gets the job done;
• prints json and text fragments to stdout, making the output unsuitable for machine analysis.

Tufin/oasdiff — ✅ ⁵

• go;
• good README;
• gets the job done;
• configurable output format (text or json);
• has a separate web service where you can compare specs online;
• actively developed by the author.

oasdiff was the one we picked as the primary tool for finding differences between OpenAPI contracts.

Here’s how to print a list of backwards-incompatible changes, for example:

oasdiff breaking \
    spec1.json \
    spec2.json

One last note: the move from one version to another is solved not just by understanding contract changes, but also by having a solid set of e2e tests.