How .NET 9.0 boosted JSON Schema performance by 32%

At endjin, we maintain Corvus.JsonSchema, an open source high-performance library for serialization and validation of JSON using JSON Schema.

Its first release was on .NET 7.0, and its performance was pretty impressive. Ian Griffiths has given a number of talks on the techniques it uses to achieve its performance goals.

Since then, the .NET 8.0 runtime shipped, and with no code changes at all, we got a "free" performance boost of ~20%.

As .NET 9.0 has launched, we are going to take a look at some benchmarks that compare with the latest build of Corvus.JsonSchema (4.0.1) running on .NET 9.0.

As you can see in the graph below, we get a performance boost of ~32% with the latest version of our codebase - 20% for free, and ~13% as a result of updating to the latest version of System.Text.Json and taking advantage of its new APIs.

If you compare with the original .NET 7 release, we're nearly 50% faster.

Another outstanding performance from the .NET runtime team.

Here are the details.

The Benchmark

We run the benchmark on a 13th Gen Intel Core i7-13800H, 1 CPU, 20 logical and 14 physical cores. .NET 9 and .NET 8 were respectively .NET Runtimes 9.0.100 and 8.0.824.36612

Validating a large array consisting of 10,000 small JSON documents. This is typical of a small JSON payload in a web API. It includes some strings, some formatted strings (e.g. email, date), and some numeric values.

{
    "name": {
        "familyName": "Oldroyd",
        "givenName": "Michael",
        "otherNames": [],
        "email": "michael.oldryoyd@contoso.com"
    },
    "dateOfBirth": "1944-07-14",
    "netWorth": 1234567890.1234567891,
    "height": 1.8
}

Note on allocations

Clearly those allocations reported by Benchmark DotNet are not "real" - you can't have 1 byte of allocation on a 64 bit .NET runtime.

We presume they come from sampling errors; the real allocations are approximately zero for this benchmark.

Notes on 7.0

.NET 7.0 is no longer supported, and we are now running on different hardware, with different characteristics in the benchmark. I have scaled the .NET 7 results to represent the current benchmark, but they should not be considered definitive.

Is this the end of the free lunch (redux)?

Well - last year, we questioned whether we would be seeing the end of the free lunch, and outlined a proposed API that would improve our performance in the absence of other runtime benefits.

These benchmarks show that, clearly we were not at the end of the free lunch!

Improvements to the code generated by the JIT have given us another ~20% performance improvement (possibly the most significant being the switch to a single GC memory barrier for struct copying, rather than individual GC barriers for each field).

Adding the JsonMarshal type gave us the "vFuture" API that we wanted. That gave us another ~13% boost. We had predicted ~15% so this was pretty close! I think the 2% discrepancy in the predicted result is simply that the baseline has shifted slightly.

Looking at the future

I anticipate further performance improvements in the .NET 10 timeframe.

We had a PR accepted on dotnet/runtime for .NET 10.0 that gives us raw access to JSON property names. Although not as immediately impactful as the value extraction, it will have implications for performance of propertyNames and patternProperties validation.

However, we can see that we are trending towards some kind of a limit - 20% YOY improvements while impressive do produce diminishing absolute returns. Unless, of course, the wizards of the JIT unlock something unexpected!