Composition, Polymorphism, and Pattern Matching with JSON Schema and .NET

At endjin, we maintain Corvus.JsonSchema, an open source high-performance library for serialization and validation of JSON using JSON Schema.

We do this because we believe that there are significant benefits to defining your data contracts using a schema language, and (contrary to popular opinion) JSON Schema is a great constraints language for generating idiomatic data types in popular developer runtimes like .NET and JavaScript.

Furthermore: one of the great features of JSON Schema is that it is based on composition. This is sometimes seen as the barrier to code generation, but we see it as a superpower.

Let's look into that in more detail.

Why can you describe JSON Schema as compositional?

A schema defines a set of constraints (e.g. the basic type, numeric ranges, patterns, constants and enumerations).

You can then compose them using keywords like $ref (the union of the constraints with the referenced schema), allOf (the union of the constraints of all the schema in the list), oneOf (the union of the constraints with exactly one of a set of schemas) and anyOf (the union of the constraints with one or more of a set of schemas).

You can really think of any JSON Schema keyword in this way. A schema is the union of all the constraints applied by all the keywords.

For example, for the purposes of constraints (notwithstanding the nuances of how references are resolved) the schema threeConstraints here:

"twoConstraints": {
  "type": "number",
  "minimum": 0
}

"threeConstraints": {
  "$ref": "twoConstraints",
  "maximum": 10
}

is equivalent to the schema threeConstraints here:

"threeConstraints": {
  "type": "number",
  "minimum": 0,
  "maximum": 10
}

Why favour composition?

Composition is a very powerful technique. Outside of the realm of JSON Schema, I often encourage developers in all languages to consider composition patterns over inheritance.

While inheritance-based polymorphism is deep in the DNA of OO languages (it being a feature of Smalltalk right from the get-go), inheritance is not the only way to deliver on the goals of polymorphism, and composition can often be a more natural way to represent a domain.

There are a number of advantages (and a few disadvantages) of composition - any or all of which may be applicable in your scenario. This is not an exhaustive list, but here are a few examples:

1. Encapsulation

   Looking at it from the point of the implementer, enscapsulation is still possible. If a person has-a name you can work on the semantics of the name without exposing implementation details to either the person, or consumers.

2. Implementation sharing

   Again, from the point of view of the implementer, composition gives you a form of implementation sharing that does not depend on a base class. For languages like C# which do not support multiple-inheritance this is a significant advantage.

3. Mitigating fragile base classes

   Fragile base class syndrome is the issue where a seemingly innocuous change to a base class can cause issues for derived classes.

   In a composition model, the composing type is really an orchestrator for the composed types. The composed types do not (under normal circumstances) make use of the composing type. Avoiding this dependency makes the system much more robust to change in both the composing type (the analog of the base class) and the composed types.

4. Separation of concerns

   From the point of view of the consumer, you can write code to operate on the name regardless of whether it appears as a property of a person or in some other context. The name client code need know nothing of the person.

5. Flexibility, polymorphism, and mix-ins

   When coupled with interfaces to define behaviour, and generics to define functions (often extension methods in C#) that operate over families of implementations, you have the ability to define extensibility points that don't depend on base-type inheritance.

   You can compose different implementations into your composing type, orchestrating the same overall behaviours through different means.

6. Testability

   The separation of concerns, and interface implementation, leads on to testability. Your unit tests (especially if you are building mocks or fakes) tend to be less fragile if you avoid complex base classes. You can write unit tests for the composed unit of functionality, irrespective of where it is used.

There is also a seventh, more subtle, but perhaps more significant point

7. Is-A is a very strong statement

   The "is-a" of inheritance is an extremely strong statement. Usually what we mean is "has the characteristics of", which, in most OO languages is best expressed through interfaces, rather than base class inheritance. Even then, you should be cautious that you have very clearly defined the semantics of your interface, or you may find that you have continued to over-state the compatibility of your types. This is even more significant when modelling data-like entities as we tend to be describing structure not behaviour. I question whether an "is-a" model is the best way to describe things that are simply structurally similar?

They are some of the common benefits - what about the disadvantages?

1. Accidental recursive composition

   We say that composition reduces risks from fragile base classes "under normal circumstances". That is because there is a danger that you unintentionally build a recursive composition graph. While there are useful applications (e.g. the Composite Pattern itself), recursive composition can re-introduce significant complexity (and, incidentally, is a common trait of poorly-performing JSON Schema).

2. OO developers are less familiar with composition

   There is a strong tendency in developer education to stress inheritance as "the way" of modelling OO systems.

   People who come from a functional background, or those who explore design patterns more deeply, tend to be more comfortable with composition, but the "inside out" nature of it can sometimes be a bit daunting for novice and experienced OO developers alike.

   Often, the first time people come across composition is through Dependency Injection or Inversion of Control patterns, and don't realise that it can be used more widely in domain and data modelling.

Union types

There is another significant opportunity when you apply composition patterns (especially when modelling data) and that is with union types.

A union type is essentially a type that may have any one of a number of different (perhaps wildly incompatible) representations.

You can typically determine which type an instance actually represents by inspecting the instance data stored for that type, in some way. Some languages have mecanisms to do that for you (often expressed as pattern matching). Others - not so much. We'll come to that later.

For example, we might want to use a union type to model the result of an operation which might fail.

This would be the union of your "successful output" (in this case a number) and your "error message" (in this case a string). When you get a result, you want to process it based on the specific type it represents from the union.

Here's some pseudo-code to express that.

type theError = string
type theOutput = number
type result = theOutput | theError

myResult = "Failure"

match myResult
  success: number => HandleSuccess()
  error: string => DealWithFailure()
  _: => WhatAreYouTalkingAbout()

JSON Schema and Union types

The oneOf and anyOf keywords in JSON Schema can be used to define a union type.

Both these keywords provide an array of schemas. They constrain the instance to match (at least) one of the schemas in the array.

The oneOf keyword requires each schema in the array to be mutually invalid (i.e. only one is permitted to be valid for any given instance).

The anyOf keyword matches at least one schema in the array (i.e. one or more of the subschema are permitted to be valid for any given instance)

It is a well-known optimization for many implementations to prefer anyOf where each subschema in the array is known to be mutually invalid, instead of oneOf. It is an efficient way of defining such a union type, because you do not need to test the other schemas once a match is found. It does, however, depend on the schema designer managing the exclusivity of the subschema. You should prefer oneOf unless you have a specific performance issue.

A common way to achieve mutually invalid schemas is to add a property to the properties keyword called e.g. tag, which is present in each subschema with a different const value. This then produces a discriminated union.

This is such a common pattern that it is codified (and complicated) in OpenAPI's JSON Schema dialect with the discriminator keyword.

.NET and Union types

But, I hear you say, C# doesn't have discriminated unions?! Well, you are quite right. It doesn't.

Although there is a proposal that is actively being worked on at the time of writing.

People often implement the JSON Schema notion of anyOf/oneOf through base-type polymorphism. They define a "discriminator" property on the base type, which you can inspect to determine the appropriate behaviour.

Notice how we are beginning to discuss things which must be resolved against a particular instance at runtime, in contrast to structure that can be defined purely statically at design time. It is important to keep this distinction in mind when you are designing your data models and APIs.

A serializer might use it to determine which particular concrete instance to instantiate from some JSON data.

Example schema with composition

{
    "type": "object",
    "required": ["myDiscriminator"],
    "properties": { "myDiscriminator": {"type": "string"} }
    "oneOf": [
        {"$ref": "#/$defs/firstType"},
        {"$ref": "#/$defs/secondType"}
    ],
    "$defs": {
        "firstType": {
            "properties": {
                "myDiscriminator": {"const": "firstType"},
                "someOtherProperty": {"type": "number" }
            }
        },
        "secondType": {
            "properties": {
                "myDiscriminator": {"const": "secondType"},
                "someOtherProperty": {"type": "string" }
            }
        }
    }
}

Hypothetical union implementation using inheritance

We can imagine that this schema was produced from code like this (or this code was generated from the schema that looks like that!).

public abstract class TheBase {
    [Discriminator]
    public abstract string MyDiscriminator { get; }
}

public class FirstType : TheBase {
    public string MyDiscriminator => "firstType";

    public double SomeOtherProperty { get; init; }
}

public class SecondType : TheBase {
    public string MyDiscriminator => "secondType";
    public string SomeOtherProperty { get; init; }
}

Some hypothetical serializer could determine that MyDiscriminator was the discriminator for that type. We have attributed the property in the base class, which is a common code first way of defining a discriminator property. Another popular solution is to auto-generate the discriminator value from the type name itself, and use a well-known property name (which can be overridden using an attribute).

If we were schema-first, we could use something like OpenAPI's discriminator keyword to do the same job.

Here's some pseudo-code that does that. Note how it uses pattern matching on the discriminator property to do the dispatch.

TheBase HypotheticalDeserialize(JsonReader hypotheticalReader)
{
    if (hyptotheticalReader.TryGetProperty("myDiscriminator", out string? value))
    {
        return value switch
        {
            null => null,
            "firstType" => FirstType.HypotheticalDeserialize(hypotheticalReader),
            "secondType" => SecondType.HypotheticalDeserialize(hypotheticalReader),
            _ => throw new SerializationException($"Unknown type {value}"),
        };
    }

    throw new SerializationException("Missing myDiscriminator property.");
}

Notice how this brings features of the dynamic world into your static structure. Effectively, that discriminator is acting as "runtime type information" that your serializer/deserializer is using to determine which types to create.

Drawbacks of unions by inheritance

This hints at a problem. You would never design a purely in-memory object model in this way (you would expect the runtime to handle this kind of dispatch for you), but it is the kind of thing that is necessary to bridge this gap between your inheritance model at runtime, and your JSON data model.

However, JSON Schema itself does not require you to model your composed schemas using a tag or discriminator in this way. As long as the schemas are mutually distinct, you will be fine. For example, you could define subschema that are { "type": "number" }, { "type": "string" } and { "type": "object" }. These are a mutually distinct set suitable for oneOf/anyOf composition.

However, they do not lend themselves naturally to base-type inheritance; their natural forms in .NET are typically value types (or the curiously-value-like String reference type).

Furthermore, typical implementations of this pattern require some kind of annotation (e.g. a .NET Attribute), or potentially conflicting assumptions about property or type names, to determine how to generate the necessary code.

How can we deal with this in a way that is consistent, produces idiomatic .NET code from JSON Schema, and requires no extensions to the standard JSON Schema dialect? That is one of the key challenges we set out to solve with Corvus.JsonSchema.

Serialization v. views over the underlying data

One of the challenges with a "serialization" model is that you are constrained by what you can (simply) express in your type system.

In the case of .NET, we typically model JSON object data as .NET objects with simple property accessors. A JSON array is transformed into some .NET collection type (typically a List<SomeType>). The other primitives map more-or-less naturally to their .NET equivalents: string and boolean become string, and bool (although the underlying JSON string is typically UTF8-encoded, while the .NET string is UTF16 under the covers). Things are a little more complex with number - there is no direct analog in .NET, though double, and decimal are commonly chosen. null is more complex still. .NET doesn't have a null type per se - just nullable versions of its own reference and value types, so some more work has to be done. This is complicated by needing to understand the difference between 'not present' in the JSON data, and explicit null.

In any transformation of this kind, there is a risk of loss of fidelity. The underlying JSON data is not necessarily exactly represented by the serializable .NET object model. There are open questions as to what to do with invalid data, or extensible data?

In serializers like Newtonsoft or NJsonSchema, we see fallbacks to generic library types when things get too complex.

Consider the schema:

{
    "type": "object",
    "properties": {
        "foo": {"type": "number"}
    }
}

The instances

{ "foo": 32, "bar": "Hello!" }

and even

{ "bar": "Hello!" }

are perfectly valid against this schema (because we are allowed additional properties by default, and foo is not a required property).

But how do we model that in our C# type model? It rapidly becomes challenging, and most code generators simply give up at this point; or ignore the unspecified properties; or fall back on the JSON object representation of their underlying library.

Views over the underlying data

System.Text.Json has a good solution to this problem. It provides lightweight views over the underlying JSON data. When you retrieve a property from a JSON object, or enumerate an array, you are provided with a stack-allocated entity such as a JsonElement or JsonProperty. This essentially wraps an index into the underlying JSON data, and gives you accessor methods to determine its primitive type, and then handle it appropriately.

This has a number of significant benefits over traditional serialization, and one massive drawback.

1. System.Text.Json is often faster, and lower-allocation

   Instead of allocating an entire, parallel object graph when loading a JSON document, it instead builds a compact index which is used to operate over the data. Typically, this backing data is all rented from a pool, avoiding unnecessary GC. The values returned as you access properties and enumerate objects and arrays are stack allocated, again avoiding GC pressure.

2. It is a pay-for-what-you-use model

   Unlike traditional serialization which converts all the underlying data to the relevant .NET types once-and-for-all, you only need to perform conversions for the values you actually use.

   Much data processing involves inspecting a small number of specific values in a larger document, and then often copying (and ultimately re-serializing) verbatim chunks of that document elsewhere.

   When coupled with the fact that lots of comparison logic can be performed efficiently and directly on the underlying UTF8 JSON bytes (e.g. string comparisons), this is a tremendous performance boost.

However, there is a huge drawback

3. There's nothing to help you with the structure of the document when writing code - no intellisense, no properties, no enumerators, no type information. None of the benefits of having .NET language features at your disposal.

Structured views with Corvus.JsonSchema

This is where Corvus.JsonSchema comes in.

Using either the command line tool generatejsonschematypes or the Source Generator Corvus.Json.SourceGenerator, we generate code that embodies the complete structure of the JSON Schema as .NET value types.

Similar to (and built on top of) System.Text.Json.JsonElement they provide zero-allocation views over the underlying JSON.

But like conventional deserialization to .NET types, they also provide you with property accessors, array enumerators, strongly-typed tuples, conversions to primitives like string, and double. Essentially, all the things you would expect if you deserialized to hand-built .NET types.

And because everything is represented as a slim .NET value type, there is no allocation cost to accessing properties, or enumerating arrays or maps.

Furthermore, there is no conversion cost to pay during deserialization. You pay for a conversion to a primitive (like a string, double, Guid, or LocalDateTime) only when you need it. In fact, you can often avoid those conversions entirely by using the built-in comparison and equality operators on e.g. JsonString that work directly with string, ReadOnlySpan<char>, and ReadOnlySpan<byte>.

That's a huge benefit in most scenarios where you would typically be deserializing to hand-crafted .NET types. But they also have a superpower - which is natural support for JSON Schema's composition model with pattern matching.

.NET union types and pattern matching

Let's look at some code generated by Corvus.JsonSchema for the example schema we looked at above.

In this case, we are using Corvus.Json.SourceGenerator but the generated code is the same if we use the CLI tool.

Here's the code that bootstraps the generator. It emits a type called Discriminated for the root schema in ./schema.json, and generates other types for all its dependencies.

namespace CompositionWalkthrough.Model;

using Corvus.Json;

[JsonSchemaTypeGenerator("./schema.json")]
public readonly partial struct Discriminated
{
}

Remember that our base schema can be oneOf two different, mutually independent schemas: firstType and secondType. The way we happen to have modelled that independence is through a discriminator property called myDiscriminator, but in fact we could have modelled this in any way we liked, as long as they were mutually invalid schemas. For example, they could have a different type keyword; or they could have different required properties.

So, as you might expect, we find that the code generation has produced additional types that represent the schemas from the discriminated union: FirstType and SecondType.

Just as with the hypothetical "hand crafted" example with base types, you can explicitly up-cast an instance of the 'base' type Discriminated to either of these discriminated union types. Here's an example - but hold on because there is a better way.

public static string ExplicitProcessingOfDiscriminatedUnion(Discriminated discriminated)
{
    return (string)discriminated.MyDiscriminator switch
    {
        "firstType" => $"First type: {((Discriminated.FirstType)discriminated).SomeOtherProperty}",
        "secondType" => $"Second type: {((Discriminated.SecondType)discriminated).SomeOtherProperty}",
        _ => $"Unrecognized match: {discriminated.MyDiscriminator}",
    };
}

As with the hand-crafted example, this code depends on your client code knowing how your union type is discriminating between the types. And you also cannot be sure that you have covered all possible cases.

It is also unnecessarily allocates a string for the switch; ideally, you would avoid this and produce comparisons directly against the underlying UTF8 JSON data.

But there is a better way. Because we know oneOf represents a composition, our code generator emits a custom pattern matching method that hides all of that explicit knowledge from the user, and ensures that you also produce complete sets of handlers for all possible members of the union (and a default fallback case).

public static string ProcessDiscriminatedUnion(Discriminated discriminated)
{
    return discriminated.Match(
        (in Discriminated.FirstType matchFirstType) => $"First type: {matchFirstType.SomeOtherProperty}",
        (in Discriminated.SecondType matchSecondType) => $"Second type: {matchSecondType.SomeOtherProperty}",
        (in Discriminated discriminated) => $"Unrecognized match: {discriminated.MyDiscriminator}");
}

Here, we use the generated Match function to do the same job.

First, note that we are no longer depending on knowledge of how the composed schemas are discriminated.

We are also provided with a complete set of possible types to implement. This is especially useful while you are evolving your schemas - the compiler will tell you where you need to update your code to handle new types, or remove old ones.

And finally, it is able to implement the discriminator in as efficient way as possible (typically without allocations, and operating over the underlying UTF8 JSON data). It doesn't depend on a particular style of discrimination - it could be a tag as with OpenAPI, or based on type, required properties or as complex a schema as you like.

In real-life code, we find the use of composition; zero-allocation views over raw JSON data; .NET native implicit and explicit conversions; and pattern matching to be such incredibly powerful and versatile tools that is well worth switching to a schema-first approach and generating code, rather than hand-crafting your data types and reverse-engineering schemas from them.

Further reading

We have published examples that illustrate the code generated by Corvus.JsonSchema for popular JSON Schema patterns. They dive deeper into some of the topics we have covered in this post. If you want to learn more, and see real, executable example code, take a look over here.