Json Schema Patterns in .NET - Mapping input and output values

In this series we are cataloging common patterns with JSON Schema and the .NET code generated by Corvus.JsonSchema.

It is especially useful for developers who want to move to a schema-first approach, but are unsure how that will map to their .NET code.

We are focusing on draft 2020-12 and the dotnet8.0 code generation, but very similar patterns apply for older versions of both .NET and draft 2019-09, draft 7, and even draft 6. We will highlight the key differences as we go.

If you have no experience of JSON Schema at all, I would recommend you read the getting started step-by-step documentation provided by the JSON Schema team.

Using conversions between arbitrary types for efficient mapping

A common problem we face in API-driven applications is the need to map data from one schema to another as you move information between the layers in your solution.

For example, it is common for the representation of an entity in your API to be similar, but not identical, to the representation of the same entity in your data store.

Or the entity in your API may be somewhat different from the same entity in a 3rd party API on which you depend to provide your service.

In .NET applications it is common to use tools like AutoMapper to help with this process.

In Corvus.JsonSchema we have a novel solution to this problem. It is possible to convert between arbitary types that implement IJsonValue<T>. You can then manipulate them appropriately to create valid instances. This works even if they are compiled into different assemblies, with no "inheritance" relationship of any kind between the types.

All of this is done with a minimum amount of copying/allocation (often truly "zero allocation") making the process highly efficient.

In this example we are going to look at 3 variations on a Customer entity which, although highly simplified, you might find in a real system.

The first is the schema we use in our API.

It has a familyName and givenName which are both string values. It also has a customerId property which is a uuid (which translates to a Guid in .NET).

File: api-customer.json

{
    "type": "object",
    "required": [ "familyName", "givenName", "customerId" ],
    "properties": {
      "familyName": {"type": "string"},
      "givenName": {"type": "string"},
      "customerId": {"type": "string", "format": "uuid"}
    }
}

The second is the type used by our corporate CRM. It represents a customer with a firstName (equivalent to our givenName) and lastName (equivalent to our familyName), along with an id that is an int64.

File: crm-customer.json

{
    "type": "object",
    "required": [ "firstName", "lastName", "id" ],
    "properties": {
      "firstName": {"type": "string"},
      "lastName": {"type": "string"},
      "id": {"type": "integer", "format": "int64"}
    }
}

The last is the type we use in our database. This has a familyName and a givenName like the string in our API, but each is constrained to a maximum length of 256 characters (presumably to set hard limits on the storage being used).

It has an id which is an int64. This represents the unique ID in the database we are using. As is common, this is intended for internal use, and not to be exposed to the outside world.

For external purposes, it also has an idDescriptors property which is an idDescriptorArray. This is, as the name implies, an array of idDescriptors. This is a discriminated union of the apiIdDescriptor, the crmIdDescriptor and a fallback genericIdDescriptor.

These idDescriptors capture the ID of the customer as it is represented in each of the systems with which we interact.

The source is another constrainedString.

File: db-customer.json

{
    "type": "object",
    "required": [ "familyName", "givenName", "idDescriptors", "id" ],
    "properties": {
        "id": {
            "type": "integer",
            "format": "int64"
        },
        "idDescriptors": { "$ref": "#/$defs/idDescriptorArray" },
        "familyName": { "$ref": "#/$defs/constrainedString" },
        "givenName": { "$ref": "#/$defs/constrainedString" }
    },

    "$defs": {
        "idDescriptorArray": {
            "type": "array",
            "minItems": 1,
            "items": { "$ref": "#/$defs/idDescriptor" }
        },
        "idDescriptor": {
            "oneOf": [
                { "$ref": "#/$defs/apiIdDescriptor" },
                { "$ref": "#/$defs/crmIdDescriptor" },
                { "$ref": "#/$defs/genericIdDescriptor" }
            ]
        },
        "apiIdDescriptor": {
            "type": "object",
            "required": [ "id", "source" ],
            "properties": {
                "id": {
                    "type": "string",
                    "format": "uuid"
                },
                "source": { "const": "api" }
            }
        },
        "crmIdDescriptor": {
            "type": "object",
            "required": [ "id", "source" ],
            "properties": {
                "id": {
                    "type": "integer",
                    "format": "int64"
                },
                "source": { "const": "crm" }
            }
        },
        "genericIdDescriptor": {
            "type": "object",
            "required": [ "id", "source" ],
            "properties": {
                "id": { "$ref": "#/$defs/constrainedString"},
                "source": { "const": "generic" }
            }
        },
        "constrainedString": {
            "type": "string",
            "maxLength": 256
        }
    }
}

This time, we are going to generate the code into different folders and namespaces for each type.

This represents the idea that they are being independently generated for each subsystem. You could build them into separate assemblies if you wished, to verify interoperability in that scenario.

Generate the code:

generatejsonschematypes --outputPath Model/CustomerApi --rootNamespace JsonSchemaSample.CustomerApi api-customer.json

Generating: ApiCustomer

generatejsonschematypes --outputPath Model/Crm --rootNamespace JsonSchemaSample.CrmApi crm-customer.json

Generating: CrmCustomer

generatejsonschematypes --outputPath Model/Database --rootNamespace JsonSchemaSample.DatabaseApi db-customer.json

Generating: GenericIdDescriptor
Generating: ApiIdDescriptor
Generating: IdDescriptorArray
Generating: ConstrainedString
Generating: DbCustomer
Generating: SourceEntity
Generating: SourceEntity
Generating: IdDescriptor
Generating: CrmIdDescriptor
Generating: SourceEntity

In this scenario we are going to imagine that we have got some updated customer details from our CRM, which is the source-of-truth for our customer name.

We are going to find the corresponding customer in our database, and update the name from the values in the CRM. If it is not valid, we will add it to our "bad data" queue for someone to deal with later.

We are then going to translate the updated customer details into a form suitable for our own API, and call our "broadcast this change to the world" API.

Example code

using Corvus.Json;
using JsonSchemaSample.CrmApi;
using JsonSchemaSample.CustomerApi;
using JsonSchemaSample.DatabaseApi;

// The Customer payload incoming from our CRM
string incomingCrmPayload =
    """
    {
        "id": 1234,
        "firstName": "Henry",
        "lastName": "James"
    }
    """;

// The customer as stored in our DB
string dbPayload =
    """
    {
        "id": 56789,
        "familyName": "Henri",
        "givenName": "James",
        "idDescriptors": [
            {"source": "crm", "id": 1234 },
            {"source": "api", "id": "169edea5-dbe5-4bf4-80a8-a97a596cb85c" }
        ]
    }
    """;

// Parse the incoming CRM payload
using var parsedCrmPayload = ParsedValue<CrmCustomer>.Parse(incomingCrmPayload);
CrmCustomer crmCustomer = parsedCrmPayload.Instance;

// Query the corresponding customer in the database
using var parsedDbCustomer = QueryDatabase(DbCustomer.CrmIdDescriptor.Create(crmCustomer.Id));
DbCustomer dbCustomer = parsedDbCustomer.Instance;

Console.WriteLine("Original customer details in DB:");
Console.WriteLine(dbCustomer);

Console.WriteLine("CRM update:");
Console.WriteLine(crmCustomer);

// Update the db customer from the CRM payload
dbCustomer = UpdateDbCustomerFromCrm(dbCustomer, crmCustomer);

WriteDatabase(dbCustomer);

Console.WriteLine("Updated database customer:");
Console.WriteLine(dbCustomer);

// Now, map the db customer back to an api customer
ApiCustomer apiCustomer = MapToApiCustomer(dbCustomer);

if (apiCustomer.IsValid())
{
    Console.WriteLine("API customer: ");
    Console.WriteLine(apiCustomer);
}
else
{
    Console.WriteLine("The updated customer was not valid for the API.");
}

ApiCustomer MapToApiCustomer(in DbCustomer dbCustomer)
{
    // We know that ApiCustomer and DbCustomer are nearly identical.
    // So we can efficiently convert the DbCustomer directly into an ApiCustomer
    // We remove the ID properties that we don't require,
    // and add the CustomerId property.
    // We have avoided copying and allocating string values - they remain as pointers
    // to the underlying UTF8 data backing the original DbCustomer instance.
    return dbCustomer
        .As<ApiCustomer>()
        .RemoveProperty(DbCustomer.JsonPropertyNames.IdDescriptorsUtf8)
        .RemoveProperty(DbCustomer.JsonPropertyNames.IdUtf8)
        .WithCustomerId(GetApiId(dbCustomer));
}

// Update the customer from the CRM value
DbCustomer UpdateDbCustomerFromCrm(in DbCustomer dbCustomer, in CrmCustomer crmCustomer)
{
    return dbCustomer
        .WithFamilyName(crmCustomer.LastName)
        .WithGivenName(crmCustomer.FirstName);
}

// Generic database query mechanism for an ID
ParsedValue<DbCustomer> QueryDatabase(in DbCustomer.IdDescriptor idDescriptor)
{
    // We imagine that this is a public entry point for our API, so we will validate
    // the type passed in to us.
    if (!idDescriptor.IsValid())
    {
        throw new InvalidOperationException("The id descriptor is not valid");
    }

    // Use pattern matching to dispatch to the appropriate handler for the id descriptor in the query
    return idDescriptor.Match(
        (in DbCustomer.ApiIdDescriptor d) => QueryWithApiId(d),
        (in DbCustomer.CrmIdDescriptor d) => QueryWithCrmId(d),
        (in DbCustomer.GenericIdDescriptor d) => throw new InvalidOperationException("Unsupported generic id"),
        (in DbCustomer.IdDescriptor d) => throw new InvalidOperationException($"Unknown ID descriptor: {d}"));
}

// Find the appropriate ID for the API in the customer object
JsonUuid GetApiId(in DbCustomer dbCustomer)
{
    foreach (DbCustomer.IdDescriptor id in dbCustomer.IdDescriptors.EnumerateArray())
    {
        // TryGetAs performs a pattern match for a single type
        if (id.TryGetAsApiIdDescriptor(out DbCustomer.ApiIdDescriptor apiId))
        {
            return apiId.Id;
        }
    }

    return JsonUuid.Undefined;
}

void WriteDatabase(in DbCustomer customer)
{
    if (!customer.IsValid())
    {
        throw new InvalidOperationException("The customer is not valid");
    }

    // Write the customer back to the database...
}

// Execute the query with the CRM ID
ParsedValue<DbCustomer> QueryWithCrmId(in DbCustomer.CrmIdDescriptor idDescriptor)
{
    // (Fake a database query for the payload)
    if (idDescriptor.Id == 1234)
    {
        return ParsedValue<DbCustomer>.Parse(dbPayload);
    }

    return DbCustomer.Undefined;
}

// Execute the query with the API ID
ParsedValue<DbCustomer> QueryWithApiId(in DbCustomer.ApiIdDescriptor idDescriptor)
{
    // (Fake a database query for the payload)
    if (idDescriptor.Id == Guid.Parse("169edea5-dbe5-4bf4-80a8-a97a596cb85c"))
    {
        return ParsedValue<DbCustomer>.Parse(dbPayload);
    }

    return DbCustomer.Undefined;
}