So far in this series, we've focused on the classic HTTP request/response pattern: a client sends a request, a server returns a response. But modern APIs increasingly need to push notifications back to their clients. They need to tell you when an order ships, when a payment succeeds, or when a long-running operation completes. OpenAPI models this with webhooks, callbacks, and links.

The notification symmetry problem

When your API server sends a webhook to a subscriber, something interesting happens to the client/server roles. Your API server becomes a client (it sends an HTTP request to the subscriber's URL). The subscriber's application becomes a server (it exposes an endpoint to receive the notification).

This role reversal creates two code generation needs from the same spec:

Both sides benefit from the same typed validation and schema enforcement that the regular openapi-client and openapi-server commands provide. The subscriber gets schema-validated payloads with typed handler interfaces. The API server gets a typed client that validates notifications before sending them.

Generating a callback server (receiving webhooks)

When your application subscribes to a service's events, you need endpoints to receive the callbacks. The openapi-callback-server command generates them:

corvusjson openapi-callback-server petstore.json \
    --rootNamespace MyApp.WebhookReceiver \
    --outputPath ./Generated/WebhookReceiver

This produces the same structure you've seen from openapi-server - handler interfaces, endpoint registration, params and result types - but for the callback and webhook operations rather than the main API paths.

Wire it up exactly as you would a regular generated server:

using Corvus.Text.Json;
using MyApp.WebhookReceiver;

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
WebApplication app = builder.Build();

WebhookHandler handler = new();
app.MapApiEndpoints(handler);

app.Run();

Your handler implements the generated interface. Each webhook or callback operation gets its own method with typed parameters:

internal sealed class WebhookHandler : IApiCallbacksHandler, IApiWebhooksHandler
{
    public ValueTask<OnEventCallbackResult> HandleOnEventCallbackAsync(
        OnEventCallbackParams parameters,
        JsonWorkspace workspace,
        CancellationToken cancellationToken = default)
    {
        // parameters.Body is the typed, schema-validated notification payload
        Console.WriteLine($"Received event: {parameters.Body.EventType}");

        return new(OnEventCallbackResult.Ok());
    }

    public ValueTask<StatusChangeWebhookResult> HandleStatusChangeWebhookAsync(
        StatusChangeWebhookParams parameters,
        JsonWorkspace workspace,
        CancellationToken cancellationToken = default)
    {
        Console.WriteLine($"Status changed to: {parameters.Body.NewStatus}");

        return new(StatusChangeWebhookResult.Ok());
    }
}

The generator separates callbacks (defined inline on path operations) from webhooks (defined at the top level of the spec) into distinct handler interfaces. If your spec defines both, your handler class implements both. If it only uses one, you only implement that interface.

Generating a callback client (sending webhooks)

On the other side, when your API server needs to dispatch notifications to subscribed clients, you generate a typed client:

corvusjson openapi-callback-client petstore.json \
    --rootNamespace MyApp.WebhookSender \
    --outputPath ./Generated/WebhookSender

This produces a client class with methods for each callback/webhook operation. You use it in your server's business logic when you need to notify subscribers:

using Corvus.Text.Json.OpenApi.HttpTransport;
using MyApp.WebhookSender;

// When a pet is adopted, notify the subscriber at their registered URL
using HttpClient httpClient = new() { BaseAddress = new Uri(subscriberCallbackUrl) };
await using HttpClientTransport transport = new(httpClient);
await using ApiWebhooksClient client = new(transport);

await using PetAdoptedWebhookResponse response = await client.PetAdoptedWebhookAsync(
    body: new PetAdoptedEvent.Source((ref PetAdoptedEvent.Builder b) =>
    {
        b.Create(petId: "pet-123"u8, adopterId: "user-456"u8);
    }));

The payload is validated against the schema before sending, just as with regular client requests. If your notification payload doesn't match the spec, you find out at publish time rather than from a confused subscriber reporting malformed data.

Runtime expressions

OpenAPI callbacks use runtime expressions to define where the callback URL comes from. A callback defined on a /subscriptions POST operation might use {$request.body#/callbackUrl} as its URL. That means the callback URL is extracted from the request body's callbackUrl field.

The code generator resolves these automatically. The generated response struct captures the context needed:

You don't write the expression resolution code yourself. The generator emits it based on what your spec declares, and the generated response types carry the captured context through so that callback invocations can resolve their bindings.

Links: hypermedia-style navigation

OpenAPI Links define follow-on operations that can be invoked from a response. They share the same runtime expression infrastructure as callbacks, but serve a different purpose: they model the "what can I do next?" question for API consumers.

When a response declares links, the generated response struct provides typed navigation methods:

// Create a pet - the 201 response links to showPetById
await using CreatePetResponse response = await client.CreatePetAsync(
    body: new NewPet.Source((ref NewPet.Builder b) =>
    {
        b.Create(name: "Luna"u8, tag: "cat"u8);
    }));

// Follow the link - petId is automatically populated from $response.body#/id
response.MatchResult(
    matchCreated: async pet =>
    {
        await using ShowPetByIdResponse petResponse =
            await response.CreatedLinks.ShowPetByIdAsync();
        // petId was extracted from the create response body automatically
        return 0;
    },
    matchDefault: error =>
    {
        Console.WriteLine($"Error: {error.Message}");
        return 0;
    });

The CreatedLinks property on the response gives you typed access to all linked operations defined for that response status. Parameters bound by runtime expressions ($response.body#/id in this case) are resolved from the captured context. You call the linked operation without manually extracting and passing the ID.

This is particularly powerful for APIs that follow HATEOAS principles, where the response tells you what actions are available next. The generated code makes link-following type-safe and parameter-free.

When callbacks and links intersect

Callbacks and links share the runtime expression mechanism, but they serve different roles:

Callbacks are about the server notifying the client asynchronously. The client registers a URL, and the server calls it later when something happens. The code generation produces both sides: a receiver (callback server) for the client, and a sender (callback client) for the server.

Links are about the client navigating to related operations synchronously. After receiving a response, the client follows a link to a related resource. The code generation adds navigation methods to the response type.

Webhooks are a special case of callbacks defined at the top level of the spec rather than inline on a specific operation. They represent spec-wide notification contracts that aren't tied to a particular request/response cycle.

In practice, many APIs use all three. A payment API might define links for navigating from a charge to its refunds (synchronous navigation), callbacks for notifying when a charge succeeds (asynchronous server-to-client), and webhooks for general platform events like account updates (top-level notifications).

Spec structure

For reference, here's how these features appear in an OpenAPI 3.2 spec:

Callbacks are defined inline on a path operation:

paths:
  /subscriptions:
    post:
      operationId: createSubscription
      callbacks:
        onEvent:
          "{$request.body#/callbackUrl}":
            post:
              operationId: onEventCallback
              requestBody:
                content:
                  application/json:
                    schema:
                      $ref: "#/components/schemas/Event"
              responses:
                "200":
                  description: Received

Webhooks are defined at the top level:

webhooks:
  statusChange:
    post:
      operationId: statusChangeWebhook
      requestBody:
        content:
          application/json:
            schema:
              $ref: "#/components/schemas/StatusChange"
      responses:
        "200":
          description: Acknowledged

Links are defined on response objects:

paths:
  /pets:
    post:
      operationId: createPet
      responses:
        "201":
          description: Created
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Pet"
          links:
            showPetById:
              operationId: showPetById
              parameters:
                petId: "$response.body#/id"

The generator processes all three forms and produces the corresponding typed code. For the complete webhook and callback reference, including the example recipes that demonstrate both sides of a webhook interaction, see the OpenAPI documentation on corvus-oss.org.

In the [ref slug=openapi-code-generation-with-corvus-filtering text=next post], we'll look at filtering - generating code for just the endpoints you need, which becomes essential when working with large API surfaces.

What's new?

This release has exactly one new feature: it fixes a long-standing problem that caused some projects to stop using Rx.NET. You could end up with 90MB of unnecessary extra build output if you had a project with both of these characteristics:

• Self-contained deployment (or Native AoT)
• A Windows-specific <TargetFramework> specifying version 10.0.19041 or later (e.g. net8.0-windows10.0.19041)

In projects like these, adding a reference to System.Reactive would cause a complete copy of the Windows Forms and WPF frameworks to be deployed as part of your application. Without trimming, this would add about 90MB to the output size. Self-contained deployment with trimming enabled was slightly less bad, adding 'just' 47MB. NativeAoT is better still, with growth of about 11MB.

Since System.Reactive.dll is around 1.5MB in size, this inflation was absurd. It was the result of a design decision made many years ago that didn't cause any of these problems at the time. Subsequent changes in the .NET ecosystem have made that decision look regrettable in hindsight.

What does this mean for me?

For most users of Rx.NET, this update will change nothing. In particular, if you aren't building for a Windows-specific target framework (e.g. if you target net10.0) there is no new or changed functionality. We have dropped support for older versions of .NET (.NET 7.0 and before) but Microsoft already stopped supporting those some time ago.

If you build for a Windows-specific target, then this change may affect you. If you were using the WPF, Windows Forms, or Windows Runtime support built into Rx.NET, you will need to add new NuGet package references. (See the next section for details.) If you were not, then you won't need to change anything, and if you are using self-contained deployment you should see the size of your built output dropping dramatically.

Breaking changes

There are three causes of breaking changes in this release:

1. we have removed support for out-of-support .NET runtimes (v7.0 and older).
2. if your project uses Rx.NET's support for WPF, Windows Forms, UWP, or Windows Runtime, you will get compiler errors after upgrading, and will need to add additional NuGet package references. (This is a compile-time breaking change only.)
3. we fixed a bug (#2247) in the nullability of OfType, which, strictly speaking, is a breaking change; the nullability now correctly reflects the (unchanged) behaviour, so this is as benign a breaking change as we think is possible.

(We saved that last bugfix for this release specifically because it is, strictly speaking, a breaking change, so for semantic versioning purposes, it needed a major version bump.)

Supported .NET versions

Rx 7.0 removes support for .NET 6.0 and .NET 7.0. Those runtimes have already been out of support for some time, but it was possible to use Rx 6.1 with them because it offered net6.0 and net6.0-windows10.0.19041 TFMs. (Rx 7.0 might actually still run on these old runtimes thanks to its netstandard2.0 support, but we do not test this so we do not support it.)

We offer net8.0, net8.0-windows10.0.19041, net472, netstandard2.0, and uap10.0.18362 targets.

We fully support running on .NET 8.0, 9.0 and 10.0. (We don't have a net10.0 target, but that's because there's nothing in net10.0 that requires code that is any different from the net8.0 code.) We will support .NET 11.0 once that ships. (It is likely that this will just mean updating the test suite, but if it turns out that there are any compatibility issues, we will ship an update. But we expect to continue offering the same targets in our NuGet packages.)

UI framework support now in separate packages

The cause of the 'bloat' problem in which System.Reactive could grow your deployable outputs by 90MB was the fact that support for WPF and Windows Forms was built right into the main System.Reactive package. To fix the problem, this functionality is available only through UI-framework-specific packages

• System.Reactive.Windows.Forms for Windows Forms
• System.Reactive.Wpf for WPF
• System.Reactive.WindowsRuntime for WinRT (e.g., CoreDispatcher) support
• System.Reactive.Uwp for UWP

Note that if you have existing code that was relying on these features being built into System.Reactive, we include a Code Analyzer in System.Reactive 7.0 that detects this and tells you exactly which NuGet package you need to be using.

Binary compatibility retained

Although we have removed WPF, Windows Forms and UI-related Windows Runtime support from the public API of System.Reactive, it does in fact remain available at runtime. This is to ensure that upgrading to Rx 7.0 doesn't break older components that were built against Rx 6.1, and which expect that functionality still to be there.

We do this by continuing to ship all the relevant code in the DLLs in the lib folder of the NuGet package. These APIs have been removed only from the ref folder. This is sufficient to prevent the 'bloat' problem while also maximizing binary compatibility.

There's one thing to be aware of: projects using the ancient packages.config mechanism don't recognize the distinction between lib and ref folders—they use the lib folder at compile time as well as runtime. So projects still using that system will continue to be able to use the WPF and Windows Forms features without adding the new package references. We do not recommend this (because we hope, many years from now, to remove that code entirely) and we do not support the use of the old packages.config mechanism with Rx.NET.

Why just this one change?

This was a surprisingly difficult problem to solve. For the full (very complex) details, you can read the 0005-package-split.md ADR, which describes the history that led up to this problem, and all the possible responses we considered.

Since this is a quite significant change in the packaging, we felt it was best to make that the focus of this release. While we have done extensive testing, and have engaged in an extensive consultation period with the community, it's possible that problems will become apparent only after more people start to use it. So we wanted to keep this change separate from anything else.

What's next?

This finally completes the goals we described in the roadmap we published back when endjin first took over maintenance of Rx.NET. That roadmap outlined what we saw as the 'must fix' issues. And although we have shipped a few new features in that time, most of the work has gone into dealing with what was essentially technical debt. We are now free to move onto more exciting work.

We are very much open to community input on what the direction should be for future work. Here are some areas we have in mind:

• Performance, especially:
  • Applying a low-allocation philosophy
  • Code generation (in other libraries we maintain, we've found code generation to be an extremely effective tool for producing very high performance code, and we have some ideas here)
  • Investigate whether we can support ref-like elements
• Consider outstanding requests for new operators, including
  • Providing operators that actually do some of the various things people imagine Throttle should do

Please try it out

This new 7.0 release of System.Reactive is available on NuGet today. If you're using Rx in your application, please try upgrading. If you have any problems, please file issues at https://github.com/dotnet/reactive/issues. Meanwhile, we hope you enjoy this new version of the Reactive Extensions for .NET.

More Rx content

As well as the two series from Carmel Eve's Rx Operators Deep Dive and Richard Kerslake's Event stream manipulation for Rx with semantic logging, you can find further information here:

• Intro to Rx.NET 3rd Edition (2025)
• Rx playlist (on the endjin YouTube channel)
• Rx 101 Workshop
• Rx talk for the dotnetsheff user group
• https://reaqtive.net/ — a persistent, reliable, distributed stream processing system based on Rx

Now let's tackle authentication. It's the part that every API needs but that most code generators either over-abstract or ignore entirely.

The authentication philosophy

Here's a design choice we made early: Corvus doesn't have its own authentication framework. There's no IAuthenticationProvider interface to implement, no built-in token cache to configure, and no proprietary middleware pipeline to learn.

Why? Because .NET already has excellent patterns for this. IHttpClientFactory with DelegatingHandler middleware is the standard composition model. Azure.Identity handles Entra ID (Azure AD) tokens. Polly handles retry on 401. These are well-documented, well-tested, and understood by every .NET developer. Adding a generator-specific auth layer on top would just be another thing to learn.

What Corvus does generate is the information you need to configure these standard patterns correctly: the OAuth2 token URL, the authorisation URL, and - crucially - the per-operation scope constants extracted from the spec's security requirements.

Generated scope constants

When your OpenAPI spec declares OAuth2 security schemes with per-operation scopes, the generator emits typed constants:

// From the spec's securitySchemes definition
IApiPetsClient.SecuritySchemes.Oauth2TokenUrl       // "https://auth.example.com/token"
IApiPetsClient.SecuritySchemes.Oauth2AuthorizationUrl // "https://auth.example.com/authorize"
IApiPetsClient.SecuritySchemes.Oauth2AvailableScopes // ["read:pets", "write:pets"]

// From the per-operation security requirements
IApiPetsClient.SecurityRequirements.ListPetsOauth2Scopes   // ["read:pets"]
IApiPetsClient.SecurityRequirements.CreatePetOauth2Scopes  // ["read:pets", "write:pets"]
IApiPetsClient.SecurityRequirements.AllOauth2Scopes        // ["read:pets", "write:pets"]

This might seem like a small thing, but it solves a real problem. In most codebases, OAuth2 scopes are hardcoded strings scattered across configuration files and handler registrations. When the API adds a new scope or changes a requirement, you find out at runtime. Worse, you request too-broad a token because it's easier than tracking per-operation scopes.

With generated constants, the scopes come from the spec. If the spec changes and you regenerate, your code either still compiles (no breaking change) or fails to build (the scope constant was renamed or removed). Either way, you stay in sync with the contract.

Microsoft Entra ID (Azure AD)

The most common pattern for Azure-hosted services. Use Azure.Identity with a standard DelegatingHandler that acquires tokens using the generated scope constants:

services.AddSingleton<TokenCredential>(
    new ClientSecretCredential(tenantId, clientId, clientSecret));

services.AddTransient<EntraTokenHandler>();
services.AddHttpClient("petstore")
    .AddHttpMessageHandler<EntraTokenHandler>();

public class EntraTokenHandler(TokenCredential credential) : DelegatingHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        AccessToken token = await credential.GetTokenAsync(
            new TokenRequestContext(IApiPetsClient.SecurityRequirements.AllOauth2Scopes),
            cancellationToken);
        request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token.Token);
        return await base.SendAsync(request, cancellationToken);
    }
}

For multi-tenant scenarios or when different operations need different privilege levels, you can request per-operation scopes:

// Read-only operation - request minimal scopes
new TokenRequestContext(IApiPetsClient.SecurityRequirements.ListPetsOauth2Scopes)

// Write operation - request elevated scopes
new TokenRequestContext(IApiPetsClient.SecurityRequirements.CreatePetOauth2Scopes)

The TokenCredential abstraction in Azure.Identity covers all the common flows - ClientSecretCredential for service-to-service, InteractiveBrowserCredential for desktop apps, DeviceCodeCredential for CLI tools, and DefaultAzureCredential for managed identity in Azure-hosted services. They all work with the same handler.

Bearer tokens (non-Entra OAuth2)

For APIs that use OAuth2 but aren't backed by Entra, the pattern is the same. Just swap the token acquisition:

public class BearerTokenHandler(ITokenService tokenService) : DelegatingHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        string token = await tokenService.GetTokenAsync(cancellationToken);
        request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token);
        return await base.SendAsync(request, cancellationToken);
    }
}

Your ITokenService can talk to Auth0, Keycloak, IdentityServer, or any other OAuth2 provider. The generated scope constants still apply. Pass them to your provider's token request.

API keys

For APIs secured with API keys, the setup is even simpler. If the key goes in a header:

services.AddHttpClient("petstore", client =>
{
    client.DefaultRequestHeaders.Add("X-Api-Key", configuration["ApiKey"]);
});

If the key goes in a query parameter, use a DelegatingHandler that appends it to the request URI:

public class ApiKeyQueryHandler(string paramName, string apiKey) : DelegatingHandler
{
    protected override Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var uriBuilder = new UriBuilder(request.RequestUri!);
        string separator = string.IsNullOrEmpty(uriBuilder.Query) ? "" : "&";
        uriBuilder.Query = uriBuilder.Query.TrimStart('?') + separator
            + Uri.EscapeDataString(paramName) + "=" + Uri.EscapeDataString(apiKey);
        request.RequestUri = uriBuilder.Uri;
        return base.SendAsync(request, cancellationToken);
    }
}

Cookie authentication

Cookie parameters declared in the spec become regular method parameters on the generated client. The generated code sets the Cookie header for you:

await using CreatePetResponse response = await petsClient.CreatePetAsync(
    session_token: "sess_k7j2m9x4"u8,
    body: new NewPet.Source((ref NewPet.Builder b) =>
    {
        b.Create(name: "Luna"u8, tag: "cat"u8);
    }));
// Wire format: Cookie: session_token=sess_k7j2m9x4

On the server side, if the cookie is declared required: true and missing from the request, the generated middleware returns 400 Problem Details automatically. Your handler is never called.

Composing auth with resilience

Because authentication is just a DelegatingHandler, you can compose it with Microsoft.Extensions.Http.Resilience (Polly v8) in the same pipeline:

services.AddHttpClient("petstore")
    .AddHttpMessageHandler<EntraTokenHandler>()
    .AddStandardResilienceHandler(options =>
    {
        options.Retry.MaxRetryAttempts = 3;
        options.Retry.BackoffType = DelayBackoffType.Exponential;
    });

This gives you exponential backoff with jitter, circuit breaking, and token refresh on 401. All of it uses standard .NET middleware that any developer on your team already understands.

What's next

In the next post, we'll look at callbacks, webhooks, and links - generating code for both sides of event notification patterns.

Now let's look at a pattern that's become increasingly common since the rise of LLM-backed APIs: streaming responses.

Why streaming matters now

A traditional REST call returns a single JSON document. The client waits, the server responds, done. But LLM completions, real-time feeds, and event-driven UIs don't work that way. They produce a sequence of items over time, and the client needs to start processing before the full response is available.

HTTP has two established patterns for this: Server-Sent Events (SSE), where each item is framed as data: {...}\n\n, and Newline-Delimited JSON (NDJSON), where each item is a single JSON line terminated by \n. Both are widely supported and work through proxies, load balancers, and CDNs.

The challenge for code generators is that these aren't regular request/response cycles. The server needs a way to push items incrementally, and the client needs to consume them as they arrive. Ideally, they do so as typed, validated documents rather than raw strings. That's what the Corvus generator handles for both sides.

Declaring a streaming response in OpenAPI

In your OpenAPI spec, a streaming endpoint looks like a regular response with text/event-stream or application/x-ndjson as the media type, plus an itemSchema that describes the shape of each individual item:

/chat:
  post:
    operationId: startVetChat
    responses:
      "200":
        content:
          text/event-stream:
            schema:
              type: array
              items:
                $ref: "#/components/schemas/ChatChunk"

The generator reads this and produces typed infrastructure on both sides.

Server: the writer callback

On the server, the generated result factory doesn't return a body directly. Instead, it accepts a writer callback. This is an async delegate that appends typed items to a stream. The generated infrastructure handles serialization and framing:

public ValueTask<StartVetChatResult> HandleStartVetChatAsync(
    StartVetChatParams parameters,
    JsonWorkspace workspace,
    CancellationToken cancellationToken = default)
{
    return new(StartVetChatResult.Ok(static async (stream, cancellationToken) =>
    {
        ChatChunk greeting = ChatChunk.ParseValue(
            """{"delta":"Hello! ","done":false}"""u8);
        await stream.AppendChatChunk(greeting, cancellationToken);

        ChatChunk answer = ChatChunk.ParseValue(
            """{"delta":"How can I help?","done":true}"""u8);
        await stream.AppendChatChunk(answer, cancellationToken);
    }));
}

Each call to AppendChatChunk serializes the item as compact JSON and writes the SSE frame (data: {"delta":"Hello! ","done":false}\n\n) to the response stream. The response stays open until the callback returns. There's no explicit "end stream" method. When you're done appending, you just return, and the generated endpoint flushes and closes the HTTP response.

If the client disconnects mid-stream, the cancellation token fires. Your callback can check it between items, or simply let the next Append call throw OperationCanceledException. Either way, you don't leak connections.

Client: IAsyncEnumerable of typed documents

On the client side, the generated response type exposes EnumerateOkItems(). It is an IAsyncEnumerable that yields pooled, typed documents as they arrive:

await using StartVetChatResponse chatResponse = await chatClient.StartVetChatAsync(
    body: new ChatRequest.Source((ref ChatRequest.Builder b) =>
    {
        b.Create(question: "My cat won't eat. What should I do?"u8);
    }));

await foreach (ParsedJsonDocument<ChatChunk> chunk in chatResponse.EnumerateOkItems())
{
    using (chunk)
    {
        Console.Write(chunk.RootElement.Delta);
    }
}

Each chunk is a pooled document. The using inside the loop returns the pooled memory after you've read the item, so you never accumulate the full stream in memory. This matters for long-running streams where thousands of items might flow through.

The generated code handles the SSE frame parsing transparently. It strips the data: prefix, handles multi-line data fields, ignores comment lines, and recognises the double-newline boundary. You just iterate typed objects.

SSE metadata: event IDs and types

Sometimes you need more than just the payload. SSE supports id: and event: fields that carry stream position and event discrimination. If you need those, use EnumerateOkSseItems() instead:

await foreach (SseItem<ChatChunk> item in chatResponse.EnumerateOkSseItems())
{
    using (item.Document)
    {
        Console.WriteLine($"Event ID: {item.Id}, Type: {item.EventType}");
        Console.Write(item.Document.RootElement.Delta);
    }
}

This is particularly useful for reconnection scenarios where you need to send a Last-Event-ID header to resume from where you left off.

NDJSON: the simpler framing

For endpoints that use application/x-ndjson, the pattern is identical from your perspective. On the server, the same writer callback appends typed items:

public ValueTask<StreamPetActivityResult> HandleStreamPetActivityAsync(
    StreamPetActivityParams parameters,
    JsonWorkspace workspace,
    CancellationToken cancellationToken = default)
{
    return new(StreamPetActivityResult.Ok(static async (stream, cancellationToken) =>
    {
        ActivityEvent checkIn = ActivityEvent.ParseValue(
            """{"eventId":"evt-1","timestamp":"2026-05-30T18:00:00Z","type":"check-in","description":"Bella checked in"}"""u8);
        await stream.AppendActivityEvent(checkIn, cancellationToken);
    }));
}

The difference is purely in the wire format: NDJSON writes {...}\n (one line per item, no data: prefix). The client-side EnumerateOkItems() works the same way, yielding typed documents one per line.

NDJSON is a better fit when you don't need SSE's reconnection semantics or event typing. It's simpler, slightly more compact, and easier to process with command-line tools like jq.

Cancellation and backpressure

The writer callback model gives you natural backpressure. Each Append call is awaitable. If the transport buffer is full because the client isn't reading fast enough, Append won't complete until there's space. You don't need to implement flow control yourself.

For long-running streams (think a real-time telemetry feed), combine the cancellation token with your data source:

return new(StreamTelemetryResult.Ok(async (stream, cancellationToken) =>
{
    await foreach (SensorReading reading in sensorService.GetReadingsAsync(cancellationToken))
    {
        await stream.AppendSensorReading(reading, cancellationToken);
    }
}));

When the client disconnects, the token cancels, the await foreach exits, the callback returns, and the response closes. Clean.

What's next

In the next post, we'll look at authentication - generated OAuth2 scope constants, Entra ID integration, API keys, and cookie-based auth, all using standard .NET middleware patterns.

Why both sides from one spec?

The whole point of a contract-first approach is that the client and server agree. If you generate only the client, you can still drift on the server side. If you generate only the server, clients can send malformed requests that pass your hand-written validation but violate the spec.

When both sides are generated from the same specification, the schema contract is enforced at both boundaries. If the client builds a request that satisfies its generated validation, the server will accept it. If either side drifts from the spec, regeneration breaks the build. That's the feedback loop we want.

Setting up the round-trip

Generate both sides with separate namespaces:

corvusjson openapi-client petstore.json \
    --rootNamespace Petstore.Client \
    --outputPath ./Generated/Client

corvusjson openapi-server petstore.json \
    --rootNamespace Petstore.Server \
    --outputPath ./Generated/Server

Start the server with ASP.NET Core minimal APIs:

using Corvus.Text.Json.OpenApi;
using Corvus.Text.Json.OpenApi.HttpTransport;
using Petstore.Server;
using Petstore.Client;

WebApplicationBuilder builder = WebApplication.CreateBuilder();
WebApplication app = builder.Build();

PetsHandler handler = new();
app.MapApiEndpoints(handler);

await app.StartAsync();
string serverUrl = app.Urls.First();

Connect the client via HttpClientTransport:

using HttpClient httpClient = new() { BaseAddress = new Uri(serverUrl) };
await using HttpClientTransport transport = new(httpClient);
await using ApiPetsClient client = new(transport);

Both sides validate against the same JSON Schema. If the client builds a valid request, the server accepts it. If either side drifts from the spec, validation fails at the boundary.

Full round-trip

Create a resource through the client, receive it from the server:

await using CreatePetResponse createResponse = await client.CreatePetAsync(
    session_token: "admin-token"u8,
    body: new NewPet.Source((ref NewPet.Builder b) =>
    {
        b.Create(
            name: "Luna"u8,
            status: "available"u8,
            tags: new NewPet.JsonStringArray.Source(
                (ref NewPet.JsonStringArray.Builder ab) =>
                {
                    ab.AddItem("friendly"u8);
                    ab.AddItem("vaccinated"u8);
                }));
    }));

createResponse.MatchResult(
    matchCreated: pet =>
    {
        Console.WriteLine($"Created: [{pet.Id}] {pet.Name}");
        return 0;
    },
    matchDefault: error =>
    {
        Console.WriteLine($"Error: {error.Message}");
        return 0;
    });

The generated server handler receives pre-validated parameters and returns typed results:

public ValueTask<CreatePetResult> HandleCreatePetAsync(
    CreatePetParams parameters,
    JsonWorkspace workspace,
    CancellationToken cancellationToken = default)
{
    return new(CreatePetResult.Created(
        body: new Pet.Source((ref Pet.Builder pb) =>
        {
            pb.Create(
                id: 42,
                name: parameters.Body.Name,
                status: parameters.Body.Status);
        }),
        workspace: workspace));
}

What's next

So far we've focused on the request/response basics. In the next post, we'll look at streaming - SSE and NDJSON responses where the server pushes a sequence of typed items and the client consumes them as IAsyncEnumerable.

Now let's flip to the server side.

The validation tax

If you've built an ASP.NET Core API by hand, you know how much code goes into not doing business logic. You parse a query parameter, check it's present, check it's the right type, check it's within range, parse the body, validate required fields, and return a 400 with a useful message if anything is off. You can easily spend more lines on input validation than on the actual operation.

And there's a worse problem: drift. The spec says limit has a maximum of 100. A developer adds a handler that doesn't enforce it. The client and server disagree about the contract, and nobody notices until a customer reports broken pagination.

We wanted a server-side story where your handler only contains business logic. All the parsing, validation, and error response generation come directly from the spec. If the spec changes, the handler's signature changes, and the compiler tells you what to fix.

What the generator produces

corvusjson openapi-server petstore.json \
    --rootNamespace Petstore.Server \
    --outputPath ./Generated

dotnet add package Corvus.Text.Json.OpenApi
dotnet add package Corvus.Text.Json

You get:

• A handler interface (IApiPetsHandler) - one async method per operation. This is the only thing you implement.
• Endpoint registration (MapApiEndpoints) - a single extension method that wires all routes with correct HTTP methods and path templates.
• Params structs - strongly-typed, already-validated request parameters and bodies.
• Result structs - factory methods for each response status code, so you can't accidentally return the wrong shape.
• Model types - the same zero-allocation models from client generation.

Your handler never validates

This is the core principle. The generated middleware runs the full validation pipeline before your handler is called:

HTTP Request arrives
  → Parse path/query/header/cookie params
  → Validate against JSON Schema
  → Parse and validate request body
  → If anything is invalid: return 400 Problem Details
  → Your Handler runs (everything is guaranteed valid)
  → Validate response body, serialize, write HTTP

If a required parameter is missing, schema validation fails, or the body can't be parsed, the request never reaches your code. The generated middleware returns a properly formatted Problem Details response. Your handler only sees valid, typed data.

Wiring it up

The setup is deliberately minimal:

using Corvus.Text.Json;
using Petstore.Server;
using Petstore.Server.Models;

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
WebApplication app = builder.Build();

PetsHandler handler = new();
app.MapApiEndpoints(handler);

app.Run();

MapApiEndpoints registers every route from the spec with the correct HTTP method and path template. You don't write app.MapGet(...) by hand. If the spec adds a new operation, the handler interface gains a new method, and the compiler tells you to implement it.

Implementing business logic

Here's what a handler looks like. Notice what's missing. You don't parse parameters, you don't validate schemas, and you don't handle protocol-level errors. You write business logic (including any business-logic errors you need to surface), and the generated infrastructure takes care of the rest:

internal sealed class PetsHandler : IApiPetsHandler
{
    public ValueTask<ListPetsResult> HandleListPetsAsync(
        ListPetsParams parameters,
        JsonWorkspace workspace,
        CancellationToken cancellationToken = default)
    {
        // parameters.Limit is already validated - guaranteed <= 100
        ListPetsResult result = ListPetsResult.Ok(
            body: new Pets.Source((ref Pets.Builder b) =>
            {
                b.AddItem(new Pet.Source((ref Pet.Builder pb) =>
                {
                    pb.Create(id: 1, name: "Luna"u8, tag: "cat"u8);
                }));
            }),
            workspace: workspace);

        return new(result);
    }
}

The JsonWorkspace provides pooled memory for building the response body. The Result factory methods mirror the spec's declared status codes. ListPetsResult.Ok(...) returns a 200, and CreatePetResult.Created(...) returns a 201. You can't accidentally return a 200 body with a 201 status.

Typed results prevent response drift

Each operation gets a result type with factory methods matching the spec:

public ValueTask<CreatePetResult> HandleCreatePetAsync(
    CreatePetParams parameters,
    JsonWorkspace workspace,
    CancellationToken cancellationToken = default)
{
    // parameters.Body.Name is guaranteed present - schema says required
    string name = (string)parameters.Body.Name;

    return new(CreatePetResult.Created(
        body: new Pet.Source((ref Pet.Builder pb) =>
        {
            pb.Create(id: 42, name: name.AsSpan(), tag: "dog"u8);
        }),
        workspace: workspace));
}

The generated code also validates your response before writing it. If your handler builds a response body that violates the output schema, you get a 500 in development. A silently malformed response does not reach clients.

What's next

In the next post, we'll wire both sides together - a generated client calling a generated server over real HTTP - and see how the contract guarantee works in practice.

Now let's apply that engine to a higher-level problem: generating HTTP clients from OpenAPI specifications.

The silent bug factory

If you've consumed a REST API from .NET, you've probably written the same kind of glue code dozens of times: constructing a URI, serializing a body, sending the request, checking the status code, then deserializing the response and hoping you got the property names right. Each of those steps is a chance for a silent bug. A misspelt query parameter won't fail at compile time. A missing required header won't surface until the server rejects it. A status code you forgot to handle won't manifest until production.

The frustrating thing is that OpenAPI specifications already contain all the information you need to get this right. The spec knows which parameters go where, what shape the body should be, and which status codes the server can return. But most code generators produce loose wrapper classes that don't enforce any of that at compile time. You still end up writing validation logic and defensive checks by hand.

We wanted a generated client that actually uses the spec as a contract. One where the compiler refuses to build if you haven't handled all the declared response codes, where parameters are validated against their JSON Schema before the request ever hits the wire, and where the generated models use the same zero-allocation pooled memory from the rest of V5.

That's what corvusjson openapi-client gives you.

Why not Kiota?

The obvious question. Microsoft Kiota is the most widely-used OpenAPI client generator for .NET, and if you need multi-language support (Java, Go, TypeScript, Python) it's the right choice.

But Kiota makes different trade-offs. It generates class-based POCO models that allocate per response. It has no schema validation. Your client happily sends a malformed request and waits for the server to reject it. It doesn't generate server stubs, so you can't use one spec to keep both sides in sync. It doesn't surface per-operation OAuth2 scopes, so you end up hardcoding scope strings. And it uses exception-based error handling for non-2xx responses, which means error paths are invisible until they throw at runtime.

Corvus takes a different position: struct-based models backed by pooled memory (12–17× less allocation than Kiota in benchmarks), built-in schema validation that's still 2–4× faster than Kiota without validation, exhaustive MatchResult for response codes enforced by the compiler, and generated server stubs from the same spec. The cost is that it's .NET only.

If you're building a .NET service where latency and allocation matter, where you want both client and server from one contract, and where the compiler should catch missing error paths, that's the Corvus sweet spot.

What comes out

The full reference documentation is on the Corvus.JsonSchema OpenAPI guide, and there's an interactive OpenAPI playground where you can paste a spec and see the generated code immediately. This post focuses on the design intent and the key patterns.

Point the generator at any OpenAPI 3.x spec (3.0, 3.1, or 3.2):

dotnet tool install --global Corvus.Json.Cli
dotnet add package Corvus.Text.Json.OpenApi.HttpTransport
dotnet add package Corvus.Text.Json

corvusjson openapi-client petstore.json \
    --rootNamespace Petstore.Client \
    --outputPath ./Generated

You get four things:

• A client class that orchestrates the full request lifecycle - parameter serialization, schema validation, HTTP transport, response parsing
• Request structs that serialize path, query, header, and cookie parameters into the correct wire format
• Response structs with exhaustive MatchResult - one handler per declared status code, enforced by the compiler
• Model types in a .Models sub-namespace - the same zero-allocation JSON Schema types used throughout V5

The key insight is that all the information needed to build a correct HTTP request is already in the spec. The generator just makes it impossible to get wrong.

Transport as an abstraction

The generated client doesn't know about HttpClient. It talks to an IApiTransport - an abstraction over how HTTP requests are sent:

using Corvus.Text.Json.OpenApi;
using Corvus.Text.Json.OpenApi.HttpTransport;
using Petstore.Client;
using Petstore.Client.Models;

using HttpClient httpClient = new() { BaseAddress = new Uri("https://petstore.example.com/v1") };
await using HttpClientTransport transport = new(httpClient);
ApiPetsClient client = new(transport);

This matters for testing. Substitute an in-memory transport that returns canned responses, without needing a running server or mocking HttpClient internals. The generated code doesn't care.

Exhaustive response handling

This is the design choice we're most opinionated about.

When an API declares multiple response codes, you should handle all of them. Not just the happy path. Most HTTP client libraries make the error path easy to ignore. A 404 slips through, an unexpected 503 crashes at runtime, and nobody notices until production.

The generated response types enforce this with MatchResult:

await using ListPetsResponse listResponse = await client.ListPetsAsync(limit: 10);

listResponse.MatchResult(
    matchOk: pets =>
    {
        foreach (Pet pet in pets.EnumerateArray())
        {
            Console.WriteLine($"[{pet.Id}] {pet.Name} (tag: {pet.Tag})");
        }

        return 0;
    },
    matchDefault: error =>
    {
        Console.WriteLine($"Error {error.Code}: {error.Message}");
        return 0;
    });

If the spec declares a 404 response and you don't provide a matchNotFound handler, the code won't compile. Same principle as exhaustive switch expressions on discriminated unions. The compiler is your safety net.

Building requests without allocations

Request bodies use the same Source and Builder pattern from the rest of V5 (if you're not familiar with this, the Corvus.JsonSchema documentation covers the mutable document model in detail). A ref-struct builder constructs the JSON object in pooled memory, so there are no intermediate objects and no heap traffic:

await using CreatePetResponse createResponse = await client.CreatePetAsync(
    body: new NewPet.Source(static (ref NewPet.Builder b) =>
    {
        b.Create(name: "Fido"u8, tag: "dog"u8);
    }));

createResponse.MatchResult(
    matchCreated: createdPet =>
    {
        Console.WriteLine($"Created: [{createdPet.Id}] {createdPet.Name}");
        return 0;
    },
    matchDefault: error =>
    {
        Console.WriteLine($"Error {error.Code}: {error.Message}");
        return 0;
    });

Required properties are mandatory parameters on Create(). Optional ones have defaults. Forget a required property? It won't compile. The schema contract is enforced at the construction site, not at serialization time.

Path parameters and typed headers

Path parameters are type-safe. The generated code URI-encodes the value and substitutes it into the path template. You never build a URL string by hand:

await using ShowPetByIdResponse showResponse = await client.ShowPetByIdAsync(petId: "pet-123"u8);

string message = showResponse.MatchResult<string>(
    matchOk: static pet => $"Found: {pet.Name}",
    matchDefault: static error => $"Not found: {error.Message}");

Response headers declared in the spec are surfaced as typed properties:

JsonString nextPage = listResponse.XNextHeader;

No more digging through response.Headers with magic strings.

Validation before the wire

Here's a subtle but important choice. The generated client validates all parameters and request bodies before sending the HTTP request.

Why? Because a cryptic 400 from the server tells you almost nothing. A client-side ArgumentException with the exact schema violation tells you everything.

try
{
    // limit: 200 exceeds the schema's "maximum: 100" constraint
    await using ListPetsResponse _ = await client.ListPetsAsync(
        limit: 200,
        validationMode: ValidationMode.Detailed);
}
catch (ArgumentException ex)
{
    Console.WriteLine($"Validation caught: {ex.Message}");
}

You control the level per-call:

Trust but verify. Catch bugs early in development, disable checks in hot paths where you know the data is valid.

What's next

In the next post, we'll flip to the server side - generating handler interfaces and ASP.NET Core endpoint registration from the same spec, so both sides of the API contract stay in sync automatically.

The Comparison

During the workshop, we loaded the same data into three different model designs in DAX Studio and compared the results:

Star Schema - Dimension tables (customers, products, dates) linked to a central fact table. Flat Table - Everything denormalised into a single wide table. Header/Detail - Two fact tables linked by a primary key.

In the example, star schema was the smallest, then flat table, then header/detail. Though it's always worth checking with your specific data!

Why Star Schema Wins

The dimension tables are small and compress well. The fact table contains foreign keys (medium cardinality) and measures. Everything benefits from good compression, and the relationships are through relatively low-cardinality keys.

Why Flat Tables Are Actually Okay

In a flat table, a lot more values are repeated. This sounds wasteful, but all those repeated customer names and product categories actually compress quite well via run-length encoding (if the sort order works out). The main downsides are that there are more columns to analyse for sort order (remember the 10-second budget), and scans over the flat table are bigger than scans over small dimension tables.

Why Header/Detail Is Usually Worst

The link between the two tables is an ID column - so with 100% cardinality, the biggest column is then repeated twice in the model. It gets almost no compression benefit. On top of that, every query traverses this high-cardinality relationship. As we discussed in the previous post: linking two large fact tables via their primary key is about the worst thing you can do.

Calculated Columns: Sometimes Helpful

Calculated columns have a bit of a bad reputation, and sometimes fairly. They increase model size and are computed during refresh. But there are cases where they can improve query performance.

Calculated columns can sometimes be a good idea, especially if the calculated column has lower cardinality than the columns it's derived from. For example, bucketing a high-precision decimal into ranges ("0-10", "10-20", etc.) creates a much lower-cardinality column that's faster to scan and filter.

But there really aren't any set rules here. Whether it helps depends on your data and queries - it always requires trying and testing.

Testing Tips

A few practical things from the workshop:

Use real data for model testing. The distribution of values matters hugely for compression, so synthetic or sampled data can give misleading results.

Enable Server Timings in DAX Studio to see how long different operations take. Scanning different columns takes different amounts of time - if you do a SUM over a column with high cardinality, it will take noticeably longer. A direct consequence of the compression mechanics.

Measure, don't guess. Your intuition about which columns are problematic isn't always right. DAX Studio shows you what's actually happening.

What's Next

That wraps up the model optimisation half of the series. Look out for the next post on how the formula engine and storage engine work together when you actually execute a DAX query.

Why Topology Is an Architectural Decision, Not an Admin Task

Workspace topology shapes your security boundaries, determines how independently teams can work, controls what you can isolate from production, and makes cost attribution either transparent or a guessing game. These are architectural problems, not admin problems.

We've seen teams struggle to refactor workspaces after they've gone live because they built everything in one space and later discovered they couldn't deploy changes safely, couldn't give data engineers access without exposing semantic layers to edit, or couldn't track costs by business domain.

Getting topology right at the start means everything else flows naturally - governance decisions inherit from it, your deployment pipeline makes sense, your access control model is simple to manage. Get it wrong and you're making exceptions, working around constraints, and potentially facing rework.

Pattern 1: The Single Workspace (and When It's Enough)

Everything lives in one workspace: your pipelines, lakehouses, dataflows, reports.

This pattern is right for:

• Early exploration and proof of concept
• A single person or a very small team
• Something that hasn't gone to production yet

The advantage is simplicity - no duplication, no complexity, no management overhead.

But the tipping point arrives very quickly, generally in one of three forms.

1. You need environment separation, because you can't test changes without risking your live data.
2. More than one team/person needs to work on the same artefacts, and their access needs diverge (engineers can't be allowed to break reports, report builders shouldn't edit pipeline logic etc.).
3. Your reports start breaking because pipeline changes weren't tested in a safe space first.

When any of these arrives, you've outgrown a single workspace. The good news is the decision to split is usually clear by that point.

Pattern 2: Environment Separation (Dev / QA / Prod)

Separate workspaces per logical environment (relating to stages of the software development lifecycle), with the same artefacts promoted through the pipeline: develop in dev, validate in QA, release to prod.

The bare minimum is two stages (dev and prod) but we'd always recommend at least three. Going straight from dev to prod is a reliable source of production incidents because someone always assumes another environment caught a problem that nobody actually tested. QA (or test, or staging) is where integration issues surface that unit tests miss.

But often, three environments isn't enough. The development/engineering/data team might need a dedicated integration environment to test their changes before sharing with business stakeholders. The business might want a dedicated pre-prod environment to validate changes against live data before promoting to prod. There might be a need for dedicated environments containing specific known sets of data for functional/load/performance testing scenarios.

The key is being clear about the purpose of each environment, and adopting a sensible and consistent naming convention. For example:

The tipping point to move beyond this pattern arrives when multiple independent use cases or business domains start sharing a workspace, because changes to one problem space affect the other, or when your access control needs diverge between them.

Pattern 3: Splitting Data Engineering and Reporting Workspaces

Separate workspaces for data engineering artefacts (pipelines, lakehouses, dataflows) and Power BI reporting, even within the same environment.

This represents a step-change from Pattern 2 because it acknowledges that these aren't the same problem. Data engineers need to move fast and experiment, but report builders need stability and a clean semantic layer. A broken pipeline shouldn't take down a exec report.

Different deployment cadences matter here too. You might deploy data engineering changes several times a day, whereas reporting changes might be monthly. Different governance boundaries follow from that - depending on your Power BI licensing situation (and the nature of your report usage), you might not even need to assign a Fabric capacity to your reporting workspaces (internal users can use existing Power BI Pro licenses).

Building on the environment naming conventions above, we now need to add another dimension to reflect the type of workspace:

The tipping point to move beyond this is when multiple independent use cases or business domains need separate governance tracks, or when noisy-neighbour capacity issues start appearing (compute from one reporting workspace competing for resources with another).

Pattern 4: Use Case or Domain Bounding

Each logical use case or business domain gets its own set of workspaces, rather than sharing a common pool.

A good way to think about this is that you're drawing security and ownership boundaries around a logic business problem and audience. Finance data engineering and supply chain data engineering are different problems, with different owners, different stakeholders, and different compliance concerns. Sharing workspaces between them creates artificial coupling.

Fabric Domains are the organisational layer that sits above this. One Domain per business unit or strategic capability, containing the workspace sets for all the use cases within that unit. This helps to unlock scaling as your Fabric deployment grows.

flowchart TD
    D1["Domain: Finance"]
    D2["Domain: Operations"]
 
    subgraph F1["Use Case: Accounts"]
        direction LR
        F1DE["DE Workspaces<br>dev / QA / prod"]
        F1RPT["Reporting Workspaces<br>dev / QA / prod"]
        F1DE --> F1RPT
    end
 
    subgraph F2["Use Case: Portfolios"]
        direction LR
        F2DE["DE Workspaces<br>dev / QA / prod"]
        F2RPT["Reporting Workspaces<br>dev / QA / prod"]
        F2DE --> F2RPT
    end
 
    subgraph O1["Use Case: Logistics"]
        direction LR
        O1DE["DE Workspaces<br>dev / QA / prod"]
        O1RPT["Reporting Workspaces<br>dev / QA / prod"]
        O1DE --> O1RPT
    end
 
    D1 --> F1
    D1 --> F2
    D2 --> O1

The tipping point is when the data within a single use case becomes large or complex enough that different layers need independent governance or are owned by different teams. At that point, your next step is usually Pattern 5.

Pattern 5: Separating Storage from Compute

The lakehouse (storage) gets its own workspace, separate from the workspace where pipelines and notebooks run.

Storage artefacts have a different lifecycle from compute artefacts. Your raw data warehouse might sit static for weeks, whereas your pipelines running against it might change daily. Separating them means you can govern them independently, upgrade compute logic without touching storage, and give different teams responsibility.

This also enables sharing. Multiple pipeline workspaces can read from a single curated lakehouse without being coupled to the engineering workspace that built it. The reporting workspaces get access to a clean Bronze/Silver/Gold layer without needing to understand the pipeline logic that produced it.

This approach also helps with cost transparency. Storage costs and compute costs separate naturally, making it clear which workload is actually expensive.

flowchart TD
    PA["Pipeline Workspace A"]
    PB["Pipeline Workspace B"]
    LH["Lakehouse Workspace<br>Shared storage"]
    RPT["Reporting Workspace"]
 
    PA -->|Write| LH
    PB -->|Write| LH
    LH -->|Curated data| RPT

The tipping point is when the data volume becomes large enough, or ownership and usage scenarios become complex enough, that different layers need independent governance and different teams managing them.

Pattern 6: Medallion Layer Splits

Bronze, Silver, and Gold layers each in their own workspace.

flowchart LR
    B["Bronze Workspace<br>Raw ingestion<br>Copy jobs"]
    S["Silver Workspace<br>Curation & cleansing<br>Pipelines / notebooks"]
    G["Gold Workspace<br>Semantic models<br>Pipelines / notebooks"]
    R["Reporting Workspace<br>Power BI / consumers"]
 
    B --> S --> G --> R

This applies when raw ingestion, curation, and the semantic reporting layer are owned by different teams or funded by different business units, when the scale of raw data creates genuine noise for downstream consumers (massive ingestion workloads getting in the way of curation), or when you want easily allow and manage access to specific layers of the medallion (e.g. providing Silver layer access for data science experiments).

There's obviously a trade off - more workspaces means more overhead, more complex deployment pipelines, more moving parts to understand and govern.

Most organisations don't need to start here, this is the pattern you reach when the simpler patterns have proven insufficient.

Choosing Your Starting Point

The patterns progress from simple to complex, but also aren't necessarily directly linear. You might need use case bounding before environment separation, or storage splitting before data engineering/reporting splitting. Knowing which to start with matters more than understanding the full spectrum.

In practice, the patterns are more of a multi-select set of options. For anything heading to production, you'll need Pattern 2 (environment separation). If you have any non-technical report builders in the picture, add the DE/reporting split (Pattern 3) from the outset. Both of these are investments that cost very little to implement early but become expensive to add later.

Our typical starting point for any engagement beyond the most trivial is a combination of patterns 2, 3, and 4. Workspaces that are bound by use case, separated by environment, and split out into data engineering and reporting.

This gives you roughly six workspaces per use case: two types (data engineering and reporting) multiplied by three environments (dev, QA, prod).

                ┌─────────────────┐  promote  ┌─────────────────┐  promote  ┌─────────────────┐
Data            │   [use-case]    │ ────────▶ │   [use-case]    │ ────────▶ │   [use-case]    │
Engineering     │   -de-dev       │           │   -de-qa        │           │   -de-prod      │
                └────────┬────────┘           └────────┬────────┘           └────────┬────────┘
                         │ feeds                       │ feeds                       │ feeds
                         ▼                             ▼                             ▼
Reporting       ┌─────────────────┐  promote  ┌─────────────────┐  promote  ┌─────────────────┐
                │   [use-case]    │ ────────▶ │   [use-case]    │ ────────▶ │   [use-case]    │
                │   -rpt-dev      │           │   -rpt-qa       │           │   -rpt-prod     │
                └─────────────────┘           └─────────────────┘           └─────────────────┘

Summary

Workspace topology in Microsoft Fabric is a foundational decision that shapes everything downstream - how you deploy changes safely, how you manage access across teams, and how clearly you can attribute costs to the workloads that generate them.

The patterns in this post represent a progression from simple to sophisticated, but they're not a linear checklist - most organisations will find their home somewhere in the middle. Understanding the obvious tipping points before you start should help you get the right foundation in place early to avoid costly rework.

This concept is important - and it has direct implications for model design. Look out for the next post where we'll put all of this together!

Relationship Cost = Key Cardinality

The cost of traversing a relationship in VertiPaq depends on the cardinality of the key column. A relationship with a small key (say, a handful of product categories) is cheap. A relationship with a high-cardinality key (say, CustomerKey across millions of customers) is expensive.

This sounds intuitive enough, but there's a subtlety that's easy to miss.

The Subtle Bit

Consider this setup:

• A Customers table with CustomerKey and Gender.
• An Orders table with OrderNumber and CustomerKey.

If you want to slice orders by Gender (low cardinality), you might think that would be cheap - Gender only has a couple of unique values, right? But to resolve that filter, the engine has to traverse the CustomerKey relationship, which is high-cardinality. The cost is determined by the relationship key, not the column you're ultimately filtering by. So it's still relatively expensive.

Why This Matters for Model Design

Now think about a header/detail model design, where you have two large tables linked by a primary key. That key has 100% cardinality - every value is unique. This means:

Every single query that crosses that relationship pays the maximum possible traversal cost. It's not just that the model is large (because the ID column is stored in both tables with no compression benefit). It's that you're also constantly paying this high traversal cost on top.

It's a double hit. Thinking about header/detail designs - basically everything has to traverse that high-order relationship, so not only is the model huge, but you're constantly paying this cost. Linking two large fact tables via their primary key is about the worst thing you can possibly do.

Fabric Weekly is a free weekly newsletter covering everything Microsoft Fabric - including:

• Data Engineering
• Reporting & Insights
• Storage & Platform
• Governance & Security
• Management & Cost
• Copilot, AI & Agents.

Whether you're just getting started with Fabric or you're deep into production workloads, there's something for you each week.

How to subscribe

You can sign up for the email newsletter (you'll need to confirm your subscription, so check your junk mail!), read every issue on the website, or subscribe to the RSS feed to be notified when a new issue lands.

Why Fabric Weekly?

Microsoft Fabric is moving fast. New features, GA announcements, and community content appear almost daily - keeping up with everything across OneLake, data pipelines, notebooks, Spark, SQL analytics endpoints, Power BI integration (and more!) is almost a full-time job in itself.

Fabric Weekly boils down all the most important updates, blog posts, videos, and community contributions into a single curated email each week - so you don't have to.

If Microsoft Fabric is part of your day job (or you're evaluating it), sign up to Fabric Weekly, or the RSS feed and keep up to date with everything going on!

And, if you have any content you'd like included, or suggestions of feeds we should keep an eye on, drop us an email at fabricweekly@endjin.com or drop a message via Bluesky at @fabricweekly.info.

Why Cardinality Drives Everything

Think back to the encoding techniques:

Hash encoding stores a dictionary of unique values. Higher cardinality means a larger dictionary, which means more memory.

Run-length encoding stores how many times values repeat in sequence. Higher cardinality means values change more frequently, so you get shorter runs and less compression.

So a column with 10 unique values and a million rows will compress brilliantly. A column with a million unique values and a million rows will barely compress at all.

And it's not just about memory. Scanning a high-cardinality column is slower too - there's simply more data to read through in the run-length encoded representation. If you do a SUM over a column with high cardinality, it will take noticeably longer than the same operation over a low-cardinality column.

Practical Ways to Reduce Cardinality

Some columns are naturally high-cardinality and you can't do much about it (primary keys, for example). But there are often easy wins hiding in your model.

Datetimes

Datetime columns are a classic culprit. A datetime with second-level precision across a few years can easily have millions of unique values. Some easy ways to reduce this:

• Split the date and time into separate columns - times are repeated each day, and dates will have far lower cardinality than the combined column.
• Reduce time precision - do you really need seconds, or would hours suffice?
• Remove the time entirely if it's not needed for your analysis.

Floating-Point Precision

Fix floating-point precision before import. A column of decimal values calculated to 15 decimal places has far higher cardinality than the same values rounded to 2 places. If your reporting doesn't need that precision, it's an easy win.

Pre-Aggregation

Group data prior to Power BI where you can. If you can aggregate in your ETL pipeline, you reduce both row count and column cardinality. This is particularly relevant if you're loading transactional data that could be summarised at a higher grain.

Finding Problem Columns

You can use DAX Studio to inspect your model size and identify "problem" columns with unexpectedly high cardinality or large memory footprints. You can do this by opening your Power BI report, opening DAX studio, and connecting to the model. If you then go on the "Advanced" tab and click "View Metrics", this will run the VertiPaq analyzer and give you your model statistics (number of columns, column sizes, cardinality, etc.).

It's worth doing this periodically - you might be surprised which columns are taking up the most space.

What's Next

So cardinality matters a lot for compression and scan speed. But it also has a major impact on something else: the cost of traversing relationships between tables. Look out for the next post where we'll dig into that.

Twenty-five years ago, work like this meant a fortune and a monolithic data warehouse. This time it is simple Python ingestion guarded by strict data contracts, a DuckDB medallion model, and requirements written in Gherkin as composable, fully tested functions, running where PySpark once failed to scale. The result: man-months of cost and "data fear" collapsed into a short laptop project, and a glimpse of how data teams freed from infrastructure can finally work as innovation teams.

Chapters

• 00:00 Introduction: endjin, DuckDB, and auditing UK energy policy
• 00:50 The data challenge and lessons from a 25-year-old data warehouse
• 01:40 Architecture: data contracts, Parquet, and a DuckDB medallion model
• 02:40 A data-driven approach with Gherkin and composable relations
• 03:40 Generating insights and why DuckDB scales where PySpark did not
• 04:30 New ways of working for data teams

Thank you very much. Thanks for having me. My name is Barry Smart. I live in Scotland and I'm director of data and AI at endjin. We are a small, fully remote technology company based in the UK, but we've got clients all around the world. We're huge fans of DuckDB. We've written a series of blog posts that share our experiences adopting DuckDB in the field. So if you're interested, please go and have a look at our blogs.

All I want to talk to you about today is a personal project that I'm working on with my dad. We've both spent a lot of time over our careers in the energy industry, and we're keen to use that experience to audit energy policy in the UK. Our concern is that the dash to Net Zero is perhaps compromising the security, reliability, and affordability of energy. Now, there's a lot of opinion out there in the general press, and we're determined to form our own opinion, taking an evidence-based approach.

The data that we need to do that is readily available. It's published by a range of government agencies, but typical of this, it's very siloed and very difficult to work with. The good news is that I've worked with this data before. It was 25 years ago at Scottish Power. We spent a fortune and a lot of man-hours building a traditional monolithic data warehouse. Some of you might be old enough to remember this kind of thing.

And whilst I could reuse the domain knowledge from that experience, I didn't want to reuse that architecture. But thankfully, in the last 25 years, thanks to Moore's Law and, more recently, DuckDB, the most powerful analytics engine I've got access to is now actually on my desktop. It's my laptop, and that's what I've used for this project.

So I've got some simple Python modules to ingest the data, apply strict data contracts to what ends up landing in Parquet format as trusted data. Then in DuckDB I've got medallion layers. Each layer is a schema in the database. Bronze is zero-copy views over the Parquet data. Silver is where all the hard work gets done to aggregate and work with the data. I materialize data in tables and then I project views into gold to generate the insights that I'm working for. So all of this is running on my laptop, but I've got confidence at some point I can push this to the cloud.

So one small technical insight I've chosen is that I've used a data-driven approach to building this solution. I've written my requirements in Gherkin syntax, and this has helped to deliver clean, self-documenting code and an extensible architecture. So what it leads to is functions that look like this. We take in a DuckDB relation type, and we return a DuckDB relation type. So it makes it a unit of functionality that's easy to test, but it also then allows me to compose those units of functionality together to build more sophisticated, complex transformations. And then when it comes to materializing these relation types, I've still got all the benefits of DuckDB with query optimization, predicate pushdown, and the like.

So what that's allowed me to do, because I've got a suite of tests, I've built the functionality incrementally all along. Every new release, every new insight I'm generating, I've got confidence that what I'm showing is right because I've got that suite of tests. We've tried to adopt this approach with PySpark in the past, and it just doesn't scale. But DuckDB just eats this for breakfast. It's a really nice way of working.

So we can generate really detailed insights like this, looking at particular points in time when there's been a major event on the network. And we've also been able to zoom out to look at — now we can detect these events — we've been able to look at the macro scale across big chunks of the data set. So we've gone from many man-months, a whole lot of money, a whole lot of pain, definitely data fear, to a project that's taken me a very short space of time to deliver, with a bit of help from Claude Code along the way, running on my laptop using DuckDB.

But I think what's more impressive about this technology is not necessarily these numbers. It's the new ways of working that it's enabling. It's allowing data teams to not worry about complex infrastructure. They can deal straight with the data, get delivering insights and responding to the feedback from users. And it's really transforming the way that we work, the way that our clients work, because they can work as innovation teams. That socio-technical system that we've been bound to for all of these years is fundamentally changing, and we can really align ourselves to the priorities and value streams within the businesses that we work with.

Thank you very much.

Now let's wrap up with migration from V4 and the production analyzers that ship with V5.

V4 isn't going away

Before we talk about migration, let's be clear: V4 is not deprecated. It continues to be maintained and is the right choice when you want the guarantees of an immutable document model.

Both engines ship in the same corvusjson CLI tool, share the same schema analysis engine, and support the same JSON Schema drafts. You choose which engine to target with the --engine flag.

Migration path

If you do want to move from V4 to V5, the path is well-supported.

Quick reference

Migration analyzers

Install the migration analyzer package:

dotnet add package Corvus.Text.Json.Migration.Analyzers

The analyzers detect V4 patterns - namespace usages, API calls, mutation patterns - and offer automatic Roslyn code fixes. In Visual Studio, you'll see lightbulb suggestions that convert V4 code to the V5 equivalent.

Copilot-assisted migration

For larger migrations, GitHub Copilot can help. The V5 documentation includes a Copilot migration guide with prompts designed to help Copilot understand the V4→V5 mapping and apply it across your codebase.

Production analyzers

V5 ships with 10 Roslyn analyzers that help you write correct, high-performance code:

Six of these have automatic code fixes. CTJ004–CTJ006 are particularly important. They catch the most common mistake with pooled memory: forgetting to dispose.

There's also CTJ-NAV, a refactoring that lets you navigate from a schema-generated type directly to its JSON Schema source in your IDE.

Getting started

# Core library (includes analyzers)
dotnet add package Corvus.Text.Json

# Source generator
dotnet add package Corvus.Text.Json.SourceGenerator

# CLI tool
dotnet tool install --global Corvus.Json.Cli

# Optional: query languages
dotnet add package Corvus.Text.Json.Jsonata
dotnet add package Corvus.Text.Json.JMESPath
dotnet add package Corvus.Text.Json.JsonLogic

# Optional: YAML
dotnet add package Corvus.Text.Json.Yaml

# Optional: JSON Patch
dotnet add package Corvus.Text.Json.Patch

# Optional: dynamic validation
dotnet add package Corvus.Text.Json.Validator

The full documentation is at github.com/corvus-dotnet/Corvus.JsonSchema.

Series recap

Over these fourteen posts, we've covered:

1. Why V5 Exists - two engines, one toolchain, different trade-offs
2. Source-Generated Types - schema-first types with full IntelliSense
3. Schema Validation - over 10× faster, all major drafts
4. Pooled-Memory Parsing - 136 bytes per document
5. Mutable Documents - builder pattern with version tracking
6. Standalone Evaluator - annotations for schema-driven tooling
7. JSONata - query and transformation with 100% conformance
8. JMESPath - on average 28× faster JSON queries
9. JsonLogic - safe business rules as data
10. YAML 1.2 - zero-allocation conversion with event streaming
11. JSON Patch - RFC 6902 with a fluent builder
12. JSON Pointer - zero-allocation path resolution
13. Extended Types - UTF-8 URIs, BigNumber, and NodaTime
14. Migration, Analyzers, and What's Next (this post)

If you have questions, find bugs, or want to contribute, the GitHub repository is the place to go.

LLMs process tokens, not bytes. Every {, }, ", and repeated property name costs tokens. Those tokens cost money and latency. TOON (Token-Oriented Object Notation) is a compact text format that preserves the JSON data model while stripping extraneous detail, and making it easier for LLMs to interpret the content.

The problem: repeated property names

Consider a 100-row array of objects - a common pattern when you feed query results or catalogue data into an LLM:

[
  {"id": 1, "name": "Alice", "score": 95},
  {"id": 2, "name": "Bob", "score": 87},
  {"id": 3, "name": "Carol", "score": 91}
]

The property names id, name, and score are repeated on every row. The braces, colons, and quotes add overhead that carries no new information after the first row.

In TOON, the same data is a table:

[3]{id,name,score}:
  1,Alice,95
  2,Bob,87
  3,Carol,91

The field list appears once. Each row is a comma-delimited value list. For arrays with many rows, the token saving is substantial.

Packages

Install with:

dotnet add package Corvus.Text.Json.Toon

or, for the lighter-weight package:

dotnet add package Corvus.Toon.SystemTextJson

Parsing TOON into a document

Parse TOON into the same pooled document model used by the rest of Corvus.Text.Json:

using Corvus.Text.Json;
using Corvus.Text.Json.Toon;

string toon = """
    name: Alice
    age: 30
    active: true
    scores[3]: 95,87,92
    """;

using ParsedJsonDocument<JsonElement> document = ToonDocument.Parse<JsonElement>(toon);
JsonElement root = document.RootElement;

Console.WriteLine($"Name:   {root.GetProperty("name").GetString()}");
Console.WriteLine($"Age:    {root.GetProperty("age").GetInt32()}");
Console.WriteLine($"Scores: {root.GetProperty("scores")}");

Output:

Name:   Alice
Age:    30
Scores: [95,87,92]

The returned ParsedJsonDocument<T> uses ArrayPool-backed memory - the same pooled lifetime model described in Part 4.

Converting TOON to JSON

When you need a JSON string (e.g. to pass to another API):

using Corvus.Text.Json.Toon;

string toon = """
    [2]{id,name,score}:
      1,Alice,95
      2,Bob,87
    """;

string json = ToonDocument.ConvertToJsonString(toon);
// [{"id":1,"name":"Alice","score":95},{"id":2,"name":"Bob","score":87}]

Converting JSON to TOON

The reverse direction detects uniform object arrays and emits them as tables automatically:

using Corvus.Text.Json.Toon;

string json = """[{"id":1,"name":"Alice","score":95},{"id":2,"name":"Bob","score":87}]""";
string toon = ToonDocument.ConvertToToonString(json);

Result:

[2]{id,name,score}:
  1,Alice,95
  2,Bob,87

Zero-allocation UTF-8 path

For hot paths, you can write TOON directly to an IBufferWriter<byte>. There is no intermediate string allocation:

using System.Buffers;
using Corvus.Text.Json.Toon;

ArrayBufferWriter<byte> buffer = new(256);
ToonDocument.ConvertToToon(
    """[{"id":1,"name":"Alice","score":95},{"id":2,"name":"Bob","score":87}]"""u8,
    buffer);

ReadOnlySpan<byte> utf8Toon = buffer.WrittenSpan;

This measured 0 B/op in benchmarks. Prefer the UTF-8 overloads whenever your input is already UTF-8 or your output destination accepts bytes.

Reader and writer options

Expanding dotted keys

By default, user.name is a literal property name. Enable path expansion to convert it into nested JSON:

using Corvus.Text.Json.Toon;

ToonReaderOptions options = new()
{
    ExpandPaths = ToonPathExpansion.Safe,
};

string json = ToonDocument.ConvertToJsonString(
    "user.name: Alice\nuser.age: 30",
    options);
// {"user":{"name":"Alice","age":30}}

Folding nested JSON keys

The reverse operation folds nested objects into dotted keys in TOON output:

using Corvus.Text.Json;
using Corvus.Text.Json.Toon;

ToonWriterOptions options = new()
{
    KeyFolding = ToonKeyFolding.Safe,
};

using ParsedJsonDocument<JsonElement> document =
    ParsedJsonDocument<JsonElement>.Parse("""{"user":{"name":"Alice"},"active":true}""");

JsonElement root = document.RootElement;
string toon = ToonDocument.ConvertToToon(in root, options);
// user.name: Alice
// active: true

All options

Error handling

Invalid TOON input throws ToonException with a 1-based line and column location:

using Corvus.Text.Json.Toon;

try
{
    ToonDocument.ConvertToJsonString("[2]: 1");
}
catch (ToonException ex)
{
    Console.WriteLine(ex.Message);
    // Reports the line and column where parsing failed
}

Corvus vs Cysharp

Cysharp's ToonEncoder is an established .NET package for encoding System.Text.Json values to TOON. The key difference: Cysharp is an encoder (JSON → TOON only), while Corvus packages are bidirectional converters. If you need to consume TOON and produce JSON, use Corvus. If you only need to serialize POCOs to TOON, Cysharp may be the simpler fit.

Benchmarks on a 100-row person array show Corvus is 1.04–1.74× faster for encoding, with the UTF-8 buffer path allocating 0 B/op compared to Cysharp's 368–648 B.

Next up

In the final post, we'll cover migration from V4, the production analyzers, and how to get started.

JSON has a deliberately simple type system - strings, numbers, booleans, null, objects, and arrays. But the data those types carry is often richer than the JSON grammar suggests. A string might be a URI. A number might have 50 significant digits. A date-time might need proper time zone handling. V5 extends the core type system with first-class support for all of these.

UTF-8 URIs and IRIs

JSON Schema defines four URI-related format keywords: uri, uri-reference, iri, and iri-reference. V5 validates and parses all four with zero-allocation ref struct types that operate directly on the UTF-8 bytes in the document buffer.

Utf8Uri and Utf8Iri

Utf8Uri is a readonly ref struct that parses a URI from a ReadOnlySpan<byte> without allocating. It gives you access to every component - scheme, authority, user, host, port, path, query, and fragment - as ReadOnlySpan<byte> slices into the original buffer:

Utf8Uri uri = Utf8Uri.CreateUri(
    "https://api.example.com:8080/v1/users?active=true#top"u8);

// Each component is a ReadOnlySpan<byte> slice - no allocation
ReadOnlySpan<byte> scheme = uri.Scheme;       // "https"
ReadOnlySpan<byte> host = uri.Host;           // "api.example.com"
ReadOnlySpan<byte> path = uri.Path;           // "/v1/users"
ReadOnlySpan<byte> query = uri.Query;         // "active=true"
ReadOnlySpan<byte> fragment = uri.Fragment;    // "top"
int port = uri.PortValue;                      // 8080

For schema-generated types with "format": "uri", the code generator emits a TryGetValue method and an explicit conversion operator:

// Schema: { "type": "string", "format": "uri" }
// Generated type: MyEndpoint

if (endpoint.TryGetValue(out Utf8UriValue uriValue))
{
    using (uriValue)
    {
        Utf8Uri uri = uriValue.Uri;
        // Access components via uri.Scheme, uri.Host, uri.Path, etc.
    }
}

// Or via explicit cast (throws FormatException if invalid)
using Utf8UriValue uriValue = (Utf8UriValue)endpoint;

Utf8UriValue is a regular (non-ref) struct that owns its backing memory. It implements IDisposable, so always use a using declaration so the backing buffer is returned to the pool.

Canonical and display forms

URIs have two standard string representations. The canonical form percent-encodes reserved characters for safe transmission. The display form decodes those sequences for human readability:

Utf8Uri uri = Utf8Uri.CreateUri(
    "https://example.com/caf%C3%A9?q=hello%20world"u8);

// Display form: decodes percent-encoded sequences for readability
// "https://example.com/café?q=hello world"
string display = uri.ToString();

// Canonical form: percent-encodes reserved characters for safe transmission
Span<byte> buffer = stackalloc byte[256];
if (uri.TryFormatCanonical(buffer, out int written))
{
    // "https://example.com/caf%C3%A9?q=hello%20world"
    ReadOnlySpan<byte> canonical = buffer.Slice(0, written);
}

// Display form as UTF-8 bytes
if (uri.TryFormatDisplay(buffer, out written))
{
    ReadOnlySpan<byte> displayUtf8 = buffer.Slice(0, written);
}

Both methods write directly to a Span<byte> with no allocation. ToString() is the convenience overload that allocates a string for the display form.

Why not System.Uri?

System.Uri merges several distinct RFC concepts into a single type. It handles absolute URIs, relative references, and IRIs all through one class, which can be confusing. A method that accepts System.Uri gives no indication of whether it expects an absolute URI, a relative reference, or an IRI. V5 separates these into distinct types (Utf8Uri, Utf8UriReference, Utf8Iri, Utf8IriReference) so the semantic intent is clear at the API boundary.

Beyond the type-safety question, System.Uri allocates a managed string and normalises the URI, which can change its representation. The Utf8 variants validate and decompose the URI in place, with no allocation and no normalisation surprises. For JSON Schema format validation, this means checking whether a string is a valid uri-reference costs nothing beyond the parse itself.

All four types are derived from the .NET runtime's own System.Uri parser, rewritten to operate on UTF-8 spans rather than managed strings.

Arbitrary-precision numerics

JSON has no precision limit on numbers. The string 99999999999999999999999999999.123456789 is perfectly valid JSON. But double gives you about 15 significant digits, and decimal gives you 28. Anything beyond that is silently truncated.

In practice, you will almost never need arbitrary-precision types. The vast majority of JSON numbers fit comfortably in int, long, double, or decimal. The right approach is to use the format keyword in your schema to bound your numeric types appropriately. Use "format": "int32", "format": "double", "format": "decimal", and so on. The code generator will then select the matching .NET type, and you get compile-time safety for free.

BigNumber and BigInteger exist for the vanishingly small number of scenarios where unbounded precision is genuinely required. That includes cryptographic values, scientific datasets with extreme precision, or financial interop where the source system sends numbers beyond 28 significant digits.

How V5 handles numbers internally

V5 never converts a JSON number to a floating-point type during validation or comparison. Instead, it parses the raw UTF-8 bytes into normalised components:

All comparison and validation operates on these components. A 500-digit JSON number is compared with perfect accuracy.

BigNumber and BigInteger

When you do need to materialise an arbitrary-precision value, there are two types. BigNumber handles decimal numbers (with a fractional part or exponent), while BigInteger handles integers of unlimited size:

using Corvus.Numerics;

// Arbitrary-precision decimal
BigNumber decimalValue = element.GetBigNumber();
BigNumber result = decimalValue * 2 + BigNumber.Parse("0.001");

// Arbitrary-precision integer
BigInteger intValue = element.GetBigInteger();

BigNumber stores a BigInteger significand and an int exponent (value = significand × 10^exponent). Both types implement INumber<T> on .NET 9+, so they work with generic math APIs.

Formatting

BigNumber implements IFormattable, ISpanFormattable, and IUtf8SpanFormattable on .NET 9+, and the static formatting methods are available on all targets including netstandard2.0. It works with string interpolation, String.Format, and direct span formatting. All the standard numeric format specifiers are supported:

BigNumber value = BigNumber.Parse("12345678901234567890.123456789");

value.ToString("G", CultureInfo.InvariantCulture);   // General: "12345678901234567890.123456789"
value.ToString("F2", CultureInfo.InvariantCulture);  // Fixed-point: "12345678901234567890.12"
value.ToString("N0", CultureInfo.InvariantCulture);  // Number with grouping: "12,345,678,901,234,567,890"
value.ToString("E3", CultureInfo.InvariantCulture);  // Scientific: "1.235E+019"
value.ToString("C", CultureInfo.GetCultureInfo("en-GB"));  // Currency: "£12,345,678,901,234,567,890.12"

For zero-allocation formatting, write directly to a UTF-8 byte span:

Span<byte> buffer = stackalloc byte[128];
if (value.TryFormat(buffer, out int bytesWritten, "F2", CultureInfo.InvariantCulture))
{
    ReadOnlySpan<byte> utf8Result = buffer.Slice(0, bytesWritten);
    // Use utf8Result directly - no string allocation
}

Extended numeric types in code generation

The code generator reads the JSON Schema format keyword to select the appropriate .NET type:

For types that are only available on modern .NET, the code generator emits #if NET guards with appropriate fallbacks.

NodaTime integration

If you work with dates and times in .NET, NodaTime is the de-facto library for rich date and time handling. It helps you think about your data more clearly and express operations on that data more precisely. V5 includes built-in UTF-8 parsers for ISO 8601 formats that produce NodaTime types directly, without going through DateTime or DateTimeOffset as an intermediate step.

When the code generator encounters these format keywords, the generated types automatically include NodaTime-typed accessors alongside the standard .NET ones:

// Generated from a schema with "format": "date-time"
OffsetDateTime when = calendarEvent.When.GetOffsetDateTime();

// The standard .NET accessor is also available
DateTimeOffset whenDto = calendarEvent.When.GetDateTimeOffset();

The parsers operate directly on the UTF-8 bytes in the document buffer. There is no intermediate string allocation. The NodaTimeExtensions namespace includes custom implementations of the Gregorian calendar calculations needed for validation, so there's no runtime dependency on the NodaTime NuGet package. The parsing is self-contained.

Next up

In the next post, we'll look at TOON - a compact text format for JSON-shaped data that reduces token count when working with LLMs.

V5 exposes this as a first-class API through Utf8JsonPointer. It lets you resolve a path against any IJsonElement<T> and get back a typed result, with no allocation.

What is JSON Pointer?

A JSON Pointer is a string that identifies a specific value in a JSON document. It uses / as a path separator, with two escape sequences: ~0 for ~ and ~1 for /.

JSON Pointer is used throughout the JSON ecosystem. JSON Patch (RFC 6902) uses it for all path and from fields. JSON Schema uses it in $ref URI fragments. OpenAPI uses it to reference components. If you work with JSON tooling, you will encounter JSON Pointer paths regularly.

Resolving a pointer

Utf8JsonPointer is a readonly ref struct that wraps a ReadOnlySpan<byte>. It validates the pointer syntax on creation and resolves it against any IJsonElement<T>:

using var doc = ParsedJsonDocument<JsonElement>.Parse("""
    {
        "store": {
            "name": "Book Shop",
            "books": [
                { "title": "JSON at Work", "price": 29.99 },
                { "title": "Schema Design", "price": 34.50 }
            ]
        }
    }
    """);

JsonElement root = doc.RootElement;

// Resolve a nested property
if (Utf8JsonPointer.TryCreateJsonPointer("/store/name"u8, out Utf8JsonPointer pointer)
    && pointer.TryResolve<JsonElement, JsonElement>(root, out JsonElement name))
{
    Console.WriteLine(name.GetString()); // "Book Shop"
}

// Resolve an array element
if (Utf8JsonPointer.TryCreateJsonPointer("/store/books/1/title"u8, out Utf8JsonPointer bookPointer)
    && bookPointer.TryResolve<JsonElement, JsonElement>(root, out JsonElement title))
{
    Console.WriteLine(title.GetString()); // "Schema Design"
}

The resolution walks the document's internal metadata table directly. There is no intermediate string allocation, no tree of path objects, and no dictionary lookup. The pointer's ReadOnlySpan<byte> slices are compared against the document's UTF-8 property names byte-by-byte.

Why a ref struct?

Utf8JsonPointer is a readonly ref struct because it holds a ReadOnlySpan<byte> that points directly into the source buffer. This means it cannot be stored on the heap, boxed, or used as a field in a class. It exists for the duration of the stack frame where you use it.

This is a deliberate design choice. A JSON Pointer is typically created, resolved, and discarded in a single operation. Making it a ref struct means zero GC pressure for this pattern.

If you need to store a pointer for later use, store the raw byte[] or string and create the Utf8JsonPointer when you need to resolve it.

Source location resolution

One of the more unusual features of Utf8JsonPointer is the ability to resolve a path to a source location in the original document. This is useful for tooling that needs to produce error messages with line numbers and column offsets.

using var doc = ParsedJsonDocument<JsonElement>.Parse("""
    {
        "name": "Alice",
        "age": 30
    }
    """);

if (Utf8JsonPointer.TryCreateJsonPointer("/age"u8, out Utf8JsonPointer pointer)
    && pointer.TryGetLineAndOffset(doc.RootElement, out int line, out int charOffset, out long lineByteOffset))
{
    Console.WriteLine($"'age' is at line {line}, column {charOffset}");
    // 'age' is at line 3, column 12
}

TryGetLineAndOffset first resolves the pointer to find the target element, then walks the document's raw UTF-8 buffer to compute the 1-based line number and character offset. This is the same mechanism that schema validation uses to produce diagnostic locations.

Segment decoding

If you need to process pointer segments individually, DecodeSegment handles the ~0 and ~1 unescaping:

ReadOnlySpan<byte> encoded = "a~1b~0c"u8; // represents "a/b~c"
Span<byte> decoded = stackalloc byte[encoded.Length];
int written = Utf8JsonPointer.DecodeSegment(encoded, decoded);
// decoded[..written] contains "a/b~c"

This is useful when you are building tooling that iterates over the segments of a pointer for custom processing, rather than resolving it against a document.

Relationship with JSON Patch

In the JSON Patch post, we showed the PatchBuilder API for applying RFC 6902 operations. Internally, every path and from field in a patch operation is resolved using Utf8JsonPointer. The two APIs share the same underlying resolution logic, so the path syntax, escaping rules, and performance characteristics are identical.

If you are building tooling that needs to inspect or manipulate JSON Patch documents programmatically, Utf8JsonPointer gives you direct access to the resolution mechanism.

Next up

In the next post, we'll look at V5's extended type system. It covers UTF-8 URIs and IRIs, arbitrary-precision numerics with BigNumber, and first-class NodaTime integration.

Now let's look at a natural companion to the mutable document model: JSON Patch.

Why JSON Patch?

APIs commonly need to accept partial updates. That means changing a subset of fields in a document without resending the whole thing. RFC 6902 JSON Patch is the standard convention for expressing those changes. It defines a JSON format for describing a sequence of operations to apply to a document, and it's widely supported across languages and frameworks.

A patch is a JSON array of operations:

[
    { "op": "replace", "path": "/name", "value": "Bob" },
    { "op": "add", "path": "/email", "value": "bob@example.com" },
    { "op": "remove", "path": "/age" }
]

Each operation specifies what to do (op), where to do it (path in RFC 6901 JSON Pointer syntax), and optionally a value or from path. All six RFC 6902 operations are supported: add, remove, replace, move, copy, and test.

Quick start

dotnet add package Corvus.Text.Json
dotnet add package Corvus.Text.Json.Patch

Build a patch with the fluent API and apply it:

using Corvus.Text.Json;
using Corvus.Text.Json.Patch;

using JsonWorkspace workspace = JsonWorkspace.Create();
using var builder = JsonDocumentBuilder<JsonElement.Mutable>.Parse(
    workspace,
    """{"name": "Alice", "age": 30}""");

JsonElement.Mutable root = builder.RootElement;

using PatchBuilder patchBuilder = root.BeginPatch(workspace)
    .Replace("/name"u8, "Bob"u8)
    .Add("/email"u8, "bob@example.com"u8)
    .Remove("/age"u8);

JsonPatchDocument patch = patchBuilder.GetPatchAndDispose();

bool success = root.TryApplyPatch(in patch);

Console.WriteLine(builder.RootElement);
// {"name":"Bob","email":"bob@example.com"}

A few things to notice here:

• BeginPatch(workspace) takes the caller's workspace. The returned JsonPatchDocument is backed by that workspace - you must keep the workspace alive for the lifetime of the patch.
• GetPatchAndDispose() finalises the patch and disposes the builder's internal resources, signalling that the builder is no longer in use. After this call, the builder must not be reused.
• The using on the builder is a safety net: if an exception is thrown during the fluent chain (before GetPatchAndDispose() is called), the using ensures the builder's resources are still cleaned up. Because Dispose() is idempotent, calling it after GetPatchAndDispose() is harmless.

Applying patches from external sources

When a patch arrives from an API request or a file and you haven't validated it, use TryValidateAndApplyPatch. This validates the patch document against the RFC 6902 JSON Schema before applying any operations:

using ParsedJsonDocument<JsonPatchDocument> patchDoc = ParsedJsonDocument<JsonPatchDocument>.Parse(
    """
    [
        { "op": "replace", "path": "/name", "value": "Charlie" },
        { "op": "add", "path": "/active", "value": true }
    ]
    """);

bool success = root.TryValidateAndApplyPatch(patchDoc.RootElement);

If you constructed the patch locally via PatchBuilder, you can skip validation and call TryApplyPatch directly.

Individual operations

Each operation is also available as a standalone extension method on JsonElement.Mutable (defined in JsonPatchExtensions), for when you need a single change without constructing a full patch document. Values accept JsonElement.Source, which has implicit conversions from string, ReadOnlySpan<byte>, bool, int, double, and many more types.

Add

Adds a value at the target path. For objects, the property is created (or replaced if it already exists). For arrays, the value is inserted at the given index. The special index - appends to the end.

root.TryAdd("/email"u8, "alice@example.com"u8);
root.TryAdd("/tags/0"u8, "important"u8);
root.TryAdd("/tags/-"u8, "new-tag"u8);

Remove

Removes the value at the target path. The target must exist.

root.TryRemove("/email"u8);
root.TryRemove("/tags/0"u8);

Replace

Replaces the value at the target path. Unlike add, the target must already exist.

root.TryReplace("/name"u8, "Bob"u8);
root.TryReplace("/age"u8, 31);

Move

Moves a value from one path to another. This is equivalent to a remove followed by an add.

root.TryMove("/old_name"u8, "/name"u8);

Copy

Copies a value from one path to another without removing the source.

root.TryCopy("/name"u8, "/display_name"u8);

Test

Tests that the value at the target path equals the expected value using deep equality. Returns true if they match. No mutation is performed.

using ParsedJsonDocument<JsonElement> expected = ParsedJsonDocument<JsonElement>.Parse("\"Alice\"");
bool matches = root.TryTest("/name"u8, expected.RootElement);

This is useful for conditional patches. If the test fails, the entire patch fails:

using PatchBuilder patchBuilder = root.BeginPatch(workspace)
    .Test("/version"u8, 1)           // guard: only apply if version is 1
    .Replace("/version"u8, 2)        // update version
    .Add("/migrated"u8, true);       // add new field

JsonPatchDocument patch = patchBuilder.GetPatchAndDispose();

bool success = root.TryApplyPatch(in patch);
// success is false if /version was not 1

JSON Pointer paths

All paths follow RFC 6901 JSON Pointer syntax:

Path-accepting methods have three overloads: ReadOnlySpan<byte> (UTF-8, preferred), ReadOnlySpan<char>, and string. For best performance, use the "..."u8 byte literal form.

Error handling

All Try* methods return bool. When TryApplyPatch returns false, operations applied before the failure are not rolled back. The document is in a partially-modified state. If you need atomic all-or-nothing semantics, take a snapshot before applying and restore it on failure. Remember CreateSnapshot() and Restore() from Part 5? That's exactly what they're for:

using var snapshot = builder.CreateSnapshot();

bool success = root.TryApplyPatch(in patch);
if (!success)
{
    builder.Restore(snapshot);
}

Next up

In the next post, we'll look at JSON Pointer resolution. It's the path syntax that JSON Patch uses under the hood, and V5 exposes it as a first-class zero-allocation API through Utf8JsonPointer.

Now let's talk about a format that sits alongside JSON in almost every modern development workflow: YAML.

YAML is everywhere

Kubernetes manifests, GitHub Actions workflows, Azure DevOps pipelines, Docker Compose files, Helm charts, OpenAPI specifications. Almost every infrastructure-as-code and CI/CD tool uses YAML as its primary configuration format.

If you validate, transform, or process configuration, you inevitably need to convert YAML to JSON. The schema validation, query languages, and processing tools all operate on JSON.

V5 includes a YAML 1.2 to JSON converter that does this with zero allocation on the hot path.

Quick start

Two packages are available:

# Full Corvus document model - when you want ParsedJsonDocument<T>, schema validation, etc.
dotnet add package Corvus.Text.Json.Yaml

# System.Text.Json only - when you want a lightweight JsonDocument, no Corvus dependencies
dotnet add package Corvus.Yaml.SystemTextJson

Parse YAML to a typed document

using Corvus.Text.Json;
using Corvus.Text.Json.Yaml;

string yaml = """
    name: Alice
    age: 30
    hobbies:
      - reading
      - cycling
    """;

using var doc = YamlDocument.Parse<JsonElement>(yaml);
JsonElement root = doc.RootElement;
Console.WriteLine(root.GetProperty("name").GetString()); // "Alice"
Console.WriteLine(root.GetProperty("age").GetInt32());    // 30

That gives you a ParsedJsonDocument<JsonElement>, the same pooled-memory document we discussed in post 4. From here you can validate against a schema, query with JMESPath or JSONata, mutate with a builder, or just read values.

Parse YAML directly to a strongly-typed element

This is where the real power of YAML-to-JSON conversion becomes clear. Because YamlDocument.Parse<T> is generic over any IJsonElement<T>, you can parse YAML directly into a schema-generated type. There is no intermediate untyped step, and the result is fully validated and strongly typed from the moment you access it.

Consider a Kubernetes Deployment manifest. You'd typically write it in YAML, but the Kubernetes API schema is published as JSON Schema. Generate your types from that schema, and then:

// Your YAML manifest - the format every Kubernetes user writes in
string manifest = """
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: web-frontend
      labels:
        app: web
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: web
      template:
        metadata:
          labels:
            app: web
        spec:
          containers:
            - name: nginx
              image: nginx:1.27
              ports:
                - containerPort: 80
    """;

// Parse directly to the generated Deployment type
using var doc = YamlDocument.Parse<Deployment>(manifest);
Deployment deployment = doc.RootElement;

// Strongly-typed access - IntelliSense, compile-time safety, no casting
string name = (string)deployment.Metadata.Name;          // "web-frontend"
int replicas = (int)deployment.Spec.Replicas;            // 3
string image = (string)deployment.Spec.Template.Spec
    .Containers[0].Image;                                // "nginx:1.27"

// Schema validation is built in
bool isValid = deployment.EvaluateSchema();

The YAML bytes flow through the tokenizer into the document's pooled memory, and you get a typed view with full IntelliSense and schema validation. The same pattern works for any schema-defined format that people author in YAML: OpenAPI specifications, GitHub Actions workflows, Helm values files, Azure Resource Manager templates, and more.

Convert to a JSON string

string json = YamlDocument.ConvertToJsonString("key: value");
Console.WriteLine(json); // {"key":"value"}

Stream to a Utf8JsonWriter

For pipeline scenarios where you're writing directly to an output buffer:

using var stream = new MemoryStream();
using var writer = new Utf8JsonWriter(stream,
    new JsonWriterOptions { Indented = true });

YamlDocument.Convert("items:\n  - one\n  - two"u8, writer);
writer.Flush();

System.Text.Json only

If you don't need the Corvus document model:

using Corvus.Yaml;

string yaml = "name: Bob\nage: 25";
using JsonDocument doc = YamlDocument.Parse(yaml);
Console.WriteLine(doc.RootElement.GetProperty("name").GetString());

It uses the same tokenizer and achieves the same conformance, without any Corvus dependency.

How it works

The converter uses a custom ref struct tokenizer that operates directly on UTF-8 bytes. There's no intermediate object model. The tokenizer emits events (scalar, sequence start, mapping start, etc.) that the converter translates directly into Utf8JsonWriter calls.

This means the hot path allocates nothing. The YAML goes in as bytes, the JSON comes out through a writer, and the only allocations are the ones Utf8JsonWriter makes for its own output buffer (which is pooled if you configure it that way).

Event streaming

The internal event model is also exposed as a public API. YamlDocument.EnumerateEvents calls your callback for each parse event, giving you zero-copy access to the raw UTF-8 data:

YamlDocument.EnumerateEvents(yamlBytes, static (in YamlEvent e) =>
{
    switch (e.Type)
    {
        case YamlEventType.Scalar:
            Console.WriteLine($"Scalar: {Encoding.UTF8.GetString(e.Value)}");
            break;
        case YamlEventType.MappingStart:
            Console.WriteLine("Object start");
            break;
        case YamlEventType.SequenceStart:
            Console.WriteLine("Array start");
            break;
    }

    return true; // continue parsing (return false to stop early)
});

Each YamlEvent is a ref struct whose spans point directly into the source buffer. The event types mirror the YAML specification: StreamStart/End, DocumentStart/End, MappingStart/End, SequenceStart/End, Scalar, and Alias. Events also carry line/column positions, anchor names, tags, and scalar styles. This is useful when you need to process YAML without converting to JSON at all. For example, you might extract specific values from a large file without parsing the whole thing.

JSON to YAML

Conversion works in both directions. YamlDocument.ConvertToYamlString takes a JSON element or raw UTF-8 JSON and produces YAML output:

string yaml = YamlDocument.ConvertToYamlString(
    """{"name": "Alice", "roles": ["admin", "user"]}""");

// name: Alice
// roles:
// - admin
// - user

There's also a streaming overload that writes to an IBufferWriter<byte> or Stream:

YamlDocument.ConvertToYaml(jsonElement, outputStream);

YamlWriterOptions controls the output format. IndentSize sets the indentation width, and SkipValidation disables structural validation for a small performance gain:

var options = new YamlWriterOptions { IndentSize = 4 };
string yaml = YamlDocument.ConvertToYamlString(json, options);

This works with both System.Text.Json.JsonElement and Corvus IJsonElement<T> types, so you can round-trip YAML through a ParsedJsonDocument. Once parsed, the document can be changed through a builder and written back out as YAML.

Utf8YamlWriter

For fine-grained control over YAML output, Utf8YamlWriter is a ref struct that writes directly to an IBufferWriter<byte> or Stream. Its API mirrors System.Text.Json.Utf8JsonWriter, so the programming model will feel familiar:

var bufferWriter = new ArrayBufferWriter<byte>();
using var writer = new Utf8YamlWriter(bufferWriter, new YamlWriterOptions { IndentSize = 2 });

writer.WriteStartMapping();
writer.WritePropertyName("name"u8);
writer.WriteStringValue("Alice"u8);
writer.WritePropertyName("roles"u8);
writer.WriteStartSequence();
writer.WriteStringValue("admin"u8);
writer.WriteStringValue("user"u8);
writer.WriteEndSequence();
writer.WriteEndMapping();

This produces:

name: Alice
roles:
  - admin
  - user

The writer supports block and flow collection styles. You can mix them in the same document. For example, use flow style for short inline sequences:

writer.WritePropertyName("tags"u8);
writer.WriteStartSequence(YamlCollectionStyle.Flow);
writer.WriteStringValue("v5"u8);
writer.WriteStringValue("release"u8);
writer.WriteEndSequence();
// tags: [v5, release]

When SkipValidation is false (the default), the writer validates structural correctness. Property names must precede values in mappings, containers must be properly closed, and you can't write a second root value. This catches mistakes at the point of the write call rather than producing silently broken output.

Schema modes

The converter supports four YAML schema modes:

var options = new YamlReaderOptions
{
    Schema = YamlSchema.Core,
    DocumentMode = YamlDocumentMode.SingleRequired,
    DuplicateKeyBehavior = DuplicateKeyBehavior.Error,
};

using var doc = YamlDocument.Parse<JsonElement>(yaml, options);

Multi-document streams

YAML supports multiple documents in a single stream, separated by ---:

---
name: Alice
---
name: Bob

Set DocumentMode = YamlDocumentMode.MultiAsArray to wrap all documents in a JSON array:

var options = new YamlReaderOptions
{
    DocumentMode = YamlDocumentMode.MultiAsArray,
};

using var doc = YamlDocument.Parse<JsonElement>(multiDocYaml, options);
// Result: [{"name":"Alice"},{"name":"Bob"}]

All YAML features

The converter supports every YAML 1.2 feature:

• Scalar styles: plain, single-quoted, double-quoted, literal block (|), folded block (>)
• Collections: block and flow sequences, block and flow mappings
• Anchors and aliases: &anchor and *alias with billion-laughs protection
• Tags: !!str, !!int, !!float, !!null, !!bool, !!seq, !!map, and custom tags
• Multi-document: --- and ... document markers
• Comments: preserved in the event stream (ignored in JSON output)

Billion-laughs protection

The YAML "billion laughs" attack uses nested anchor/alias expansion to create exponentially large documents from tiny input. The converter enforces two configurable limits:

var options = new YamlReaderOptions
{
    MaxAliasExpansionDepth = 64,         // Default
    MaxAliasExpansionSize = 1_000_000,   // Default - max nodes from alias expansion
};

Expansion that exceeds either limit throws a YamlException.

Conformance

The converter passes 100% of the JSON-testable cases in the yaml-test-suite. That means 279 valid and 94 error cases (373 of 402 total). The remaining 29 cases exercise YAML features with no JSON equivalent (complex keys, empty keys, bare tags) and don't provide JSON reference output.

Next up

In the next post, we'll look at JSON Patch. It provides RFC 6902 support with a fluent builder that operates directly on the mutable document model.

Now let's complete the query language trilogy with something a bit different: a rule engine.

Rules as data

Here's a common problem. You have business logic that changes frequently - discount calculations, eligibility checks, risk scoring, feature flags - and every time the rules change, you redeploy.

What if the rules were data? Stored in a database, versioned, auditable, and evaluated safely without executing arbitrary code?

That's the job of JsonLogic.

A rule is a JSON object:

{"if": [
    {">": [{"var": "age"}, 65]},
    "senior",
    {">=": [{"var": "age"}, 18]},
    "adult",
    "minor"
]}

You evaluate it against data:

{"age": 42}

To produce a result: "adult".

Where schema validation ends and business rules begin

JSON Schema tells you whether data is structurally valid; it has the correct types, required fields present, values within range. It answers "is this a well-formed order?" But it doesn't answer "does this customer qualify for a discount?" or "should this claim be escalated?" That's business logic, and it changes on a different cadence from your data contracts.

Schema validation and rule evaluation are complementary: you validate the structure first with EvaluateSchema(), then apply business rules with JsonLogic to data you already know is well-formed.

Quick start

dotnet add package Corvus.Text.Json
dotnet add package Corvus.Text.Json.JsonLogic

String in, string out:

using Corvus.Text.Json.JsonLogic;

string? result = JsonLogicEvaluator.Default.EvaluateToString(
    """{"+":[{"var":"a"},{"var":"b"}]}""",
    """{"a":3,"b":4}""");

Console.WriteLine(result); // "7"

Zero-allocation evaluation with a workspace:

using Corvus.Text.Json;
using Corvus.Text.Json.JsonLogic;

using var ruleDoc = ParsedJsonDocument<JsonElement>.Parse(
    """{"+":[{"var":"a"},{"var":"b"}]}""");
using var dataDoc = ParsedJsonDocument<JsonElement>.Parse(
    """{"a":3,"b":4}""");

using JsonWorkspace workspace = JsonWorkspace.Create();

JsonLogicRule rule = new(ruleDoc.RootElement);
JsonElement result = JsonLogicEvaluator.Default.Evaluate(
    rule, dataDoc.RootElement, workspace);

Console.WriteLine(result.GetRawText()); // "7"

The evaluator compiles the rule into a delegate tree on first use and caches it. Subsequent evaluations of the same rule skip compilation entirely.

Real-world example: discount rules

Say you have a pricing service that applies discounts based on customer attributes:

{
    "if": [
        {"and": [
            {">=": [{"var": "order.total"}, 100]},
            {"==": [{"var": "customer.tier"}, "gold"]}
        ]},
        {"*": [{"var": "order.total"}, 0.85]},
        {">=": [{"var": "order.total"}, 50]},
        {"*": [{"var": "order.total"}, 0.95]},
        {"var": "order.total"}
    ]
}

This rule says: gold customers with orders over £100 get 15% off; everyone else with orders over £50 gets 5% off; the rest pay full price. The rule can live in your database and be loaded at startup, ready for evaluation on each request.

using var ruleDoc = ParsedJsonDocument<JsonElement>.Parse(discountRuleJson);
JsonLogicRule rule = new(ruleDoc.RootElement);

// For each request:
using JsonWorkspace workspace = JsonWorkspace.Create();
JsonElement result = JsonLogicEvaluator.Default.Evaluate(
    rule, orderData, workspace);
decimal discountedTotal = result.GetDecimal();

Change the rules? Update the database. No redeployment.

Custom operators

You can extend the rule set with custom operators:

var evaluator = new JsonLogicEvaluator(
    customOperators: new Dictionary<string, IOperatorCompiler>
    {
        ["dayOfWeek"] = DayOfWeekCompiler
    });

Custom operators can override built-in operators too. This is useful when you need domain-specific semantics for common operations.

Source generator

For rules known at build time, the source generator eliminates all runtime compilation:

using Corvus.Text.Json.JsonLogic;

namespace MyApp.Rules;

[JsonLogicRule("Rules/discount-rule.json")]
internal static partial class DiscountRule;

The generated code constant-folds literal expressions. A rule like {"merge": [[1,2], [3,4]]} is pre-computed at compile time and evaluates in 12 nanoseconds.

CLI code generation

For rules managed outside the build pipeline:

corvusjson jsonlogic Rules/discount-rule.json \
    --className DiscountRule \
    --namespace MyApp.Rules \
    --outputPath Generated/

If your rules use custom operators, you can supply a .jlops file:

corvusjson jsonlogic Rules/discount-rule.json \
    --className DiscountRule \
    --namespace MyApp.Rules \
    --outputPath Generated/ \
    --operators Rules/custom-operators.jlops

The generated code is identical to what the source generator produces.

Performance

All benchmarks compare against JsonEverything (the established .NET JsonLogic implementation):

Time comparison (selected scenarios)

Memory comparison

The runtime evaluator is faster than JsonEverything in 18 of 19 scenarios, with a geometric mean of approximately 3× faster (0.22× JE ratio means ~4.5× on average for the winning scenarios).

When to choose JsonLogic vs JSONata

JsonLogic is the right tool when the rules themselves need to be portable data, stored in a database, edited by non-developers, and versioned independently of code. JSONata is the right tool when you need more expressive power for data transformation.

Next up

In the next post, we'll look at YAML 1.2 support. It provides zero-allocation conversion from YAML to JSON with 100% conformance.

Now let's look at JMESPath. It's a simpler, standardised alternative where the performance story is, frankly, quite something.

What is JMESPath?

JMESPath is a query language for JSON. You've likely already used it. It's the expression language behind aws --query, az --query, and jp (the JMESPath CLI). It supports path navigation, projections, filtering, slicing, multiselect, pipe expressions, and built-in functions like sort, sum, min, max, join, and contains.

Unlike JSONata, JMESPath is deliberately limited. There is no arithmetic, no user-defined functions, and no side effects. It's a pure query language with a standardised specification and a cross-implementation conformance test suite.

The Corvus implementation passes all 892 test cases. It achieves 100% conformance.

Quick start

dotnet add package Corvus.Text.Json
dotnet add package Corvus.Text.Json.JMESPath

This is the simplest approach, with a cloned result and no workspace management:

using Corvus.Text.Json;
using Corvus.Text.Json.JMESPath;

using ParsedJsonDocument<JsonElement> document = ParsedJsonDocument<JsonElement>.Parse("""
    {
        "locations": [
            {"name": "Seattle", "state": "WA"},
            {"name": "New York", "state": "NY"},
            {"name": "Bellevue", "state": "WA"},
            {"name": "Olympia", "state": "WA"}
        ]
    }
    """u8);

JsonElement result = JMESPathEvaluator.Default.Search(
    "locations[?state == 'WA'].name | sort(@) | {WashingtonCities: join(', ', @)}",
    document.RootElement);

Console.WriteLine(result);
// {"WashingtonCities":"Bellevue, Olympia, Seattle"}

That expression filters an array by state, extracts the name field, sorts alphabetically, and constructs a new object with a joined string. All of that happens in a single pipeline.

Zero-allocation evaluation

For production use with controlled lifetime:

using Corvus.Text.Json;
using Corvus.Text.Json.JMESPath;

using var dataDoc = ParsedJsonDocument<JsonElement>.Parse(jsonData);
using JsonWorkspace workspace = JsonWorkspace.Create();

JsonElement result = JMESPathEvaluator.Default.Search(
    "people[?age > `30`].name",
    dataDoc.RootElement,
    workspace);

The workspace pools all intermediate memory. The expression is compiled and cached on first use. Subsequent calls with the same expression are zero-allocation.

Performance

Here's where JMESPath gets interesting. All benchmarks use JmesPath.Net as the baseline:

Across all 21 benchmarks, the geometric mean speedup is approximately 28× for the interpreted runtime. Almost every scenario shows zero allocation. That compares with 3–80 KB per query for JmesPath.Net.

The speedups come from several architectural choices:

1. No per-expression object model - JmesPath.Net parses expressions into an intermediate AST with per-node allocations. Corvus compiles to delegate trees with pooled workspace memory.
2. Pipe fusion - consecutive pipe stages are fused into a single pass where possible.
3. UTF-8 throughout - property comparisons operate on raw UTF-8 bytes, avoiding transcoding.
4. Source-generator optimisation - the code generator eliminates delegate dispatch entirely, which is why CG times are often even lower.

Source generator

When expressions are known at build time:

using Corvus.Text.Json.JMESPath;

namespace MyApp.Expressions;

[JMESPathExpression("Expressions/active-people.jmespath")]
public static partial class ActivePeopleExpression;

The expression file is registered as an AdditionalFiles item in your project.

CLI code generation

For expressions managed outside the build pipeline:

corvusjson jmespath Expressions/active-people.jmespath \
    --className ActivePeopleQuery \
    --namespace MyApp.Queries \
    --outputPath Generated/

This reads the expression file, generates an optimized static C# class, and writes it to the output path. The generated code is identical to what the source generator produces.

When to choose JMESPath vs JSONata

If your team already uses JMESPath through the AWS or Azure CLI, it's the natural choice for JSON querying in your .NET code. If you need more expressive power, JSONata is the right tool. That includes arithmetic, string manipulation, and custom functions.

Next up

In the next post, we'll complete the query language trilogy with JsonLogic - a safe, side-effect-free rule engine for evaluating business rules stored as JSON.

Now we're moving on to query languages. V5 includes three, and we'll start with JSONata, the most expressive of the trio.

What is JSONata?

JSONata is a functional query and transformation language for JSON. It gives you path navigation, filtering, higher-order functions ($map, $filter, $reduce, $sort), object construction, string manipulation, arithmetic, regex, and user-defined functions. It's Turing-complete, and it's the standard expression language for tools like Node-RED and IBM Cloud Pak.

The Corvus implementation passes all 1,665 official test suite cases. It achieves 100% conformance on both .NET 10.0 and .NET Framework 4.8.1.

Quick start

Install the packages:

dotnet add package Corvus.Text.Json
dotnet add package Corvus.Text.Json.Jsonata

The simplest approach is string in, string out:

using Corvus.Text.Json.Jsonata;

string? result = JsonataEvaluator.Default.EvaluateToString(
    "FirstName & ' ' & Surname",
    """{"FirstName": "Fred", "Surname": "Smith", "Age": 28}""");

Console.WriteLine(result); // "Fred Smith"

That's three lines of code to evaluate a JSONata expression, with no document model, workspace management, or disposal to worry about.

Zero-allocation evaluation

For production code where you control the lifetime of parsed documents, use the full API:

using Corvus.Text.Json;
using Corvus.Text.Json.Jsonata;

using var dataDoc = ParsedJsonDocument<JsonElement>.Parse(
    """
    {
        "FirstName": "Fred",
        "Surname": "Smith",
        "Age": 28,
        "Address": { "City": "London" }
    }
    """);

JsonElement result = JsonataEvaluator.Default.Evaluate(
    "FirstName & ' ' & Surname",
    dataDoc.RootElement);

Console.WriteLine(result.GetRawText()); // "Fred Smith"

On subsequent calls with the same expression, the compiled delegate tree is cached internally, so you get zero-allocation evaluation from the second call onward.

If you want explicit control over the workspace that pools intermediate memory, pass one in:

using JsonWorkspace workspace = JsonWorkspace.Create();

JsonElement result = JsonataEvaluator.Default.Evaluate(
    "FirstName & ' ' & Surname",
    dataDoc.RootElement,
    workspace);

Practical example: reshaping API responses

JSONata really shines when you need to reshape data. Say you get a complex employee record from an API and need a simpler structure:

{
    "name": FirstName & " " & Surname,
    "mobile": Contact.Phone[type = "mobile"].number,
    "city": Address.City
}

This navigates nested objects, filters arrays by a predicate, and constructs a new shape. It all happens in a single expression. In C#:

string expression = """
    {
        "name": FirstName & " " & Surname,
        "mobile": Contact.Phone[type = "mobile"].number,
        "city": Address.City
    }
    """;

string? result = JsonataEvaluator.Default.EvaluateToString(
    expression,
    employeeJson);

Evaluating to a strongly-typed result

EvaluateToString is convenient for interop with existing code, but the real power is evaluating directly to a generated JSON type. The generic Evaluate<T> overload returns the result as any schema-generated type:

// ContactSummary is generated from a JSON Schema:
// { "type": "object", "properties": {
//     "name": { "type": "string" },
//     "mobile": { "type": "string" },
//     "city": { "type": "string" }
// }}

using var dataDoc = ParsedJsonDocument<JsonElement>.Parse(employeeJson);
using JsonWorkspace workspace = JsonWorkspace.Create();

ContactSummary summary = JsonataEvaluator.Default.Evaluate<ContactSummary>(
    expression,
    dataDoc.RootElement,
    workspace);

// Strongly-typed access - no casting, no string parsing
string name = (string)summary.Name;
string city = (string)summary.City;

// Schema validation is built in
bool isValid = summary.EvaluateSchema();

The JSONata expression reshapes the data, and the generated type gives you compile-time safety over the result. You can validate the output against its schema, pass it to other V5 APIs, mutate it with a builder, or serialize it directly.

Three modes of evaluation

1. Interpreted (runtime)

Best when expressions are determined at runtime. Use this for user-supplied queries, configuration-driven transforms, or any scenario where the expression isn't known at compile time.

2. Source generator

When expressions are known at build time, the source generator compiles them to optimized static C#:

using Corvus.Text.Json.Jsonata;

namespace MyApp.Expressions;

[JsonataExpression("Expressions/full-name.jsonata")]
public static partial class FullNameExpression;

The expression file (full-name.jsonata) contains the JSONata expression and is registered as an AdditionalFiles item in your project. The generated code eliminates delegate dispatch and enables constant folding. Pure arithmetic like 1 + 2 * 3 is folded to a literal value at compile time, giving sub-nanosecond "evaluation."

3. CLI code generation

For expressions managed outside the build pipeline:

corvusjson jsonata Expressions/full-name.jsonata \
    --className FullNameExpression \
    --namespace MyApp.Expressions \
    --outputPath Generated/

Performance

Here are some representative benchmarks comparing the Corvus interpreted runtime, code-generated evaluator, and Jsonata.Net.Native v3.0.0 (the reference .NET implementation). All measured on .NET 10.0:

Employee transform (multi-step expression)

{
    "name": Employee.FirstName & " " & Employee.Surname,
    "mobile": Contact.Phone[type = "mobile"].number
}

The interpreted evaluator is 1.8× faster with 90% less allocation. Code-gen is 1.9× faster with 97% less allocation.

Property navigation and math

Across 32 benchmarks, the geometric mean speedup is approximately 2× for the interpreted evaluator, with 90–100% less memory in almost every scenario. The memory reduction is the stronger headline. Most benchmarks show zero allocation where the reference implementation allocates 1–10 KB.

Try it in your browser

The JSONata Playground lets you experiment with expressions using the Corvus interpreted runtime - no installation needed.

Next up

In the next post, we'll look at JMESPath - a simpler, standardised query language used by AWS CLI and Azure CLI, where the performance story is even more dramatic.

Now let's look at a different code generation mode. You'll want this if you're building schema-driven tooling.

Not every tool needs a full type system

The source generator we covered in post 2 produces strongly-typed C# models: property accessors, validation, serialization, mutable builders, the works. That's exactly what you want when you're building an application that uses JSON data.

But some tools don't use the data. They describe it. A form generator reads the schema's title, description, and default keywords to render input fields. A documentation tool extracts examples and deprecated flags. A configuration editor shows readOnly and writeOnly hints.

These tools need annotations, not types.

What are JSON Schema annotations?

The JSON Schema specification defines a set of keywords that are annotation-producing. They carry metadata about the data rather than constraints on it:

These annotations flow through the validation process according to the specification. For example, annotations from a then branch are only collected if the if condition passes. Annotations from composition keywords (allOf, anyOf, oneOf) are merged according to the spec rules.

Getting this right requires a fully compliant evaluator. The V5 standalone evaluator provides exactly that.

Generating the standalone evaluator

With the source generator

Set EmitEvaluator = true on the attribute to generate both the typed model and the standalone evaluator:

using Corvus.Text.Json;

namespace MyApp.Models;

[JsonSchemaTypeGenerator("Schemas/person.json", EmitEvaluator = true)]
public readonly partial struct Person;

With the CLI tool

To generate only the evaluator (no typed models):

corvusjson jsonschema Schemas/person.json \
    --rootNamespace MyApp.Evaluators \
    --outputPath Generated/ \
    --codeGenerationMode SchemaEvaluationOnly

Or generate both:

corvusjson jsonschema Schemas/person.json \
    --rootNamespace MyApp.Models \
    --outputPath Generated/ \
    --codeGenerationMode Both

The standalone evaluator is a single static class, considerably smaller than the full type hierarchy. It supports the same schema drafts (4, 6, 7, 2019-09, 2020-12, OpenAPI 3.0) and the same validation semantics.

Collecting annotations

To collect annotations, run the evaluator in Verbose mode and use JsonSchemaAnnotationProducer:

using var doc = ParsedJsonDocument<JsonElement>.Parse(jsonText);
JsonElement instance = doc.RootElement;

// Validate in Verbose mode - this collects all annotations
using var collector = JsonSchemaResultsCollector.Create(
    JsonSchemaResultsLevel.Verbose);
instance.EvaluateSchema(collector);

// Enumerate annotations - zero-allocation ref struct enumerator
foreach (JsonSchemaAnnotationProducer.Annotation annotation
    in JsonSchemaAnnotationProducer.EnumerateAnnotations(collector))
{
    Console.WriteLine(
        $"  {annotation.GetInstanceLocationText()} " +
        $"[{annotation.GetKeywordText()}] " +
        $"= {annotation.GetValueText()}");
}

Each Annotation has four pieces of information:

• Instance location - the JSON Pointer to the value being annotated (e.g., "" for root, "/name" for a property)
• Keyword - which annotation keyword produced it (e.g., "title", "description")
• Schema location - where in the schema the annotation was defined
• Value - the raw JSON value of the annotation

Writing annotations as JSON

For structured output, WriteAnnotationsTo produces a JSON object grouped by instance location, then keyword, then schema location. This is useful when feeding the output into other tools, such as a form renderer:

using var collector = JsonSchemaResultsCollector.Create(
    JsonSchemaResultsLevel.Verbose);
instance.EvaluateSchema(collector);

using var buffer = new MemoryStream();
using (var writer = new Utf8JsonWriter(buffer,
    new JsonWriterOptions { Indented = true }))
{
    JsonSchemaAnnotationProducer.WriteAnnotationsTo(collector, writer);
}

This produces output like:

{
    "": {
        "title": {
            "#": "\"Person\""
        },
        "description": {
            "#": "\"A person with a name and optional age\""
        }
    },
    "/name": {
        "title": {
            "#/properties/name": "\"Full name\""
        }
    }
}

A form generator can walk this structure and render appropriate inputs for each annotated property. It can use each title as a label alongside the description as help text, pre-fill defaults where provided, and apply readOnly/writeOnly constraints to control editability.

When to use which mode

Next up

We've now covered the core V5 model - types, validation, pooled memory, mutation, and annotations. In the next post, we'll start looking at the query and transformation languages, beginning with JSONata.

Now let's look at the other side of the coin: mutation.

The V4/V5 trade-off

This is the fundamental design decision in V5, so it's worth being explicit about it.

In V4, every document is immutable. If you want to change a property, you call a With*() method that returns a new instance with the modification applied. The old instance is unchanged. This is the functional approach. It is safe, thread-friendly, and easy to reason about.

V4 is smarter than a naive copy-on-write: it avoids copying unmodified parts of the document and defers serialization to write operations. But it still has to create new immutable data structures wherever objects or arrays are modified. In a pipeline where you parse, modify, and write JSON repeatedly, that adds up to a lot of short-lived allocations.

V5 takes the opposite approach. JsonDocumentBuilder lets you mutate documents in place, using pooled memory managed by a JsonWorkspace. You create a workspace, build or modify documents, write the output, and dispose everything. The workspace recycles the memory for the next operation.

This is the builder approach. It is fast and low-allocation, but you need to be mindful of lifetimes and ownership. V5 includes version tracking to catch stale references at runtime, which mitigates the most common class of bugs.

Neither approach is universally better. If you want safety guarantees, use V4. If you want maximum throughput, use V5.

The workspace

Every mutable operation starts with a JsonWorkspace:

using JsonWorkspace workspace = JsonWorkspace.Create();

The workspace manages pooled buffers and Utf8JsonWriter instances. When you dispose it, all resources go back to the pool. You can create multiple builders within a single workspace. They share the pooled resources.

Building from scratch

The most common pattern is building an object using a builder delegate:

using JsonWorkspace workspace = JsonWorkspace.Create();

using var doc = JsonElement.CreateBuilder(
    workspace,
    new(static (ref objectBuilder) =>
    {
        objectBuilder.AddProperty("name"u8, "Alice"u8);
        objectBuilder.AddProperty("age"u8, 30);
        objectBuilder.AddProperty("active"u8, true);
    }));

Console.WriteLine(doc.RootElement.ToString());
// {"name":"Alice","age":30,"active":true}

That new(...) is a target-typed new. The compiler knows it needs a JsonElement.Source from the CreateBuilder parameter type. The static modifier on the delegate prevents accidental closure allocations. Use UTF-8 string literals (u8) for property names to avoid transcoding overhead.

What is Source?

JsonElement.Source is a ref struct that acts as a discriminated union. It can hold any value that might appear in a JSON document. It has implicit conversions from over 30 .NET types, so you rarely need to think about it:

• Primitives: bool, int, long, double, decimal, float, short, byte, and the unsigned variants, plus Half, Int128, UInt128
• Strings: string, ReadOnlySpan<char>, ReadOnlySpan<byte> (UTF-8)
• Dates: DateTime, DateTimeOffset, and NodaTime types (LocalDate, OffsetDateTime, Period, etc.)
• Other: Guid, Uri, BigNumber, BigInteger, JsonElement, JsonElement.Mutable
• Delegates: JsonElement.ObjectBuilder.Build for nested objects, JsonElement.ArrayBuilder.Build for nested arrays

This is what makes the builder API feel natural. JsonElement.ObjectBuilder.AddProperty has direct overloads for all these types, so objectBuilder.AddProperty("age"u8, 30) just works. The int matches directly. For CreateBuilder and SetProperty, your value implicitly converts to a Source. Either way, you just pass values. The type system handles the rest.

For objects and arrays, you pass a builder delegate:

// Object builder - delegate receives ref JsonElement.ObjectBuilder
objectBuilder.AddProperty("address"u8, static (ref addressBuilder) =>
{
    addressBuilder.AddProperty("city"u8, "London"u8);
});

// Array builder - delegate receives ref JsonElement.ArrayBuilder
objectBuilder.AddProperty("tags"u8, static (ref tagsBuilder) =>
{
    tagsBuilder.AddItem("admin"u8);
    tagsBuilder.AddItem("user"u8);
});

There's also a generic Source<TContext> variant for passing context to a delegate without allocating a closure - useful in hot paths where even a single delegate allocation matters.

Nested objects and arrays

Builder delegates compose naturally:

using var doc = JsonElement.CreateBuilder(
    workspace,
    new(static (ref objectBuilder) =>
    {
        objectBuilder.AddProperty("user"u8, static (ref userBuilder) =>
        {
            userBuilder.AddProperty("name"u8, "Alice"u8);
            userBuilder.AddProperty("roles"u8, static (ref rolesBuilder) =>
            {
                rolesBuilder.AddItem("admin"u8);
                rolesBuilder.AddItem("editor"u8);
            });
        });
    }));

// {"user":{"name":"Alice","roles":["admin","editor"]}}

Parse-and-mutate

In most real-world scenarios, you're receiving JSON, modifying it, and sending it on. Parse directly into a mutable builder for the best performance:

using JsonWorkspace workspace = JsonWorkspace.Create();

// Single pass - UTF-8 bytes become the builder's backing store
using var builder = JsonDocumentBuilder<JsonElement.Mutable>.Parse(
    workspace,
    """{"status":"pending","count":5}""");

JsonElement.Mutable root = builder.RootElement;
root.SetProperty("status", "completed"u8);
root.SetProperty("count", 10);

Console.WriteLine(root.ToString());
// {"status":"completed","count":10}

All the same Parse overloads are available - from strings, UTF-8 bytes, streams, or a Utf8JsonReader.

Retaining the original

If you need to keep an immutable copy alongside the mutable version - for auditing, comparison, or read-only queries - use the two-step approach:

using JsonWorkspace workspace = JsonWorkspace.Create();

using var sourceDoc = ParsedJsonDocument<JsonElement>.Parse(
    """{"name":"Original","value":100}""");

// Convert to mutable - sourceDoc remains unchanged
using var builder = sourceDoc.RootElement.CreateBuilder(workspace);

builder.RootElement.SetProperty("name", "Modified"u8);

Console.WriteLine(sourceDoc.RootElement.ToString());  // {"name":"Original",...}
Console.WriteLine(builder.RootElement.ToString());     // {"name":"Modified",...}

Version tracking

Here's how V5 catches the aliasing bugs that in-place mutation can introduce.

Every JsonDocumentBuilder tracks a ulong version number. When you obtain a mutable element reference, it captures the current version. If you modify the document through a different reference and then try to use the stale one, V5 throws InvalidOperationException:

JsonElement.Mutable root = builder.RootElement;
JsonElement.Mutable name = root.GetProperty("name");

// Mutate through root - this bumps the version
root.SetProperty("name", "Changed"u8);

// Try to use the stale reference
name.GetString();  // throws InvalidOperationException

This won't catch every possible misuse, but it catches the most common class of bugs: holding a reference across a mutation boundary.

Clone and freeze

When you're working with a mutable document, you sometimes need an immutable copy. It is a value that won't go stale when you mutate the builder again. Freeze() gives you exactly that.

Freeze() performs a fast blit of the metadata and value backing arrays into a new immutable document registered in the same workspace, without a serialization round-trip. The result is immutable, and you can keep mutating the original builder while the frozen element stays valid:

using JsonWorkspace workspace = JsonWorkspace.Create();
using var doc = ParsedJsonDocument<JsonElement>.Parse("""{"name": "Alice", "age": 30}""");
using var builder = doc.RootElement.CreateBuilder(workspace);

builder.RootElement.SetProperty("age"u8, 31);

// Freeze - cheap immutable copy, stays in the workspace
JsonElement frozen = builder.RootElement.Freeze();

// Keep mutating - the frozen element is unaffected
builder.RootElement.SetProperty("age"u8, 99);

Assert.Equal(31, frozen.GetProperty("age"u8).GetInt32()); // still 31

The frozen element is tied to the workspace's lifetime. It's backed by pooled memory and will be cleaned up with the workspace.

On the other hand, if you need data to escape the workspace entirely (e.g. to return it to a caller who shouldn't need to worry about workspaces or lifetimes) then that's what Clone() is for.

Clone() serializes the mutable element into a fresh immutable ParsedJsonDocument that owns its own memory on the GC heap. The clone is completely independent of both the builder and the workspace, so it remains valid after both are disposed:

JsonElement clone;

using (JsonWorkspace workspace = JsonWorkspace.Create())
using (var doc = ParsedJsonDocument<JsonElement>.Parse("[[[]]]"))
using (var builder = doc.RootElement.CreateBuilder(workspace))
{
    clone = builder.RootElement[0].Clone();
    // builder and workspace are disposed here
}

// clone is still valid - it owns its own memory
Assert.Equal("[[]]", clone.GetRawText());

Use Freeze() for cheap immutable copies within the workspace scope. Use Clone() when the result needs to escape entirely.

Snapshots for rollback

JsonDocumentBuilderSnapshot<T> captures the complete state of a builder so you can restore it later. This is useful for speculative mutations where you may want to roll back if something goes wrong. We mentioned this briefly in the context of JSON Patch, but it's a general-purpose mechanism.

using JsonWorkspace workspace = JsonWorkspace.Create();
using var doc = ParsedJsonDocument<JsonElement>.Parse("""{"status": "pending", "retries": 0}""");
using var builder = doc.RootElement.CreateBuilder(workspace);

// Take a snapshot before applying changes
using var snapshot = builder.CreateSnapshot();

builder.RootElement.SetProperty("status"u8, "processing");
builder.RootElement.SetProperty("retries"u8, 1);

// Something went wrong - roll back to the snapshot
builder.Restore(snapshot);

Assert.Equal("pending", builder.RootElement.GetProperty("status"u8).GetString());
Assert.Equal(0, builder.RootElement.GetProperty("retries"u8).GetInt32());

CreateSnapshot() creates a rented copy of the builder's internal state, and Restore() copies it back. The snapshot is IDisposable and must be disposed to return the rented buffers to the pool.

Dynamic construction with runtime data

Real-world JSON isn't all static strings. Here's how you mix structure with runtime data:

string[] tags = ["admin", "user", "active"];

using JsonWorkspace workspace = JsonWorkspace.Create();

using var doc = JsonElement.CreateBuilder(
    workspace,
    new((ref objectBuilder) =>
    {
        objectBuilder.AddProperty("id"u8, Guid.NewGuid());

        // Runtime collection becomes a JSON array
        objectBuilder.AddProperty("tags"u8, (ref tagsBuilder) =>
        {
            foreach (string tag in tags)
            {
                tagsBuilder.AddItem(tag);
            }
        });
    }));

The delegate in this example captures tags from the enclosing scope, so it can't be static. That's often fine, but if you need to avoid the closure allocation there is a Source<TContext> overload that lets you pass context explicitly:

string[] tags = ["admin", "user", "active"];

using JsonWorkspace workspace = JsonWorkspace.Create();

using var doc = JsonElement.CreateBuilder(
    workspace,
    tags,
    static (in string[] tags, ref JsonElement.ObjectBuilder objectBuilder) =>
    {
        objectBuilder.AddProperty("id"u8, Guid.NewGuid());

        objectBuilder.AddProperty("tags"u8, tags,
            static (in string[] tags, ref JsonElement.ArrayBuilder tagsBuilder) =>
            {
                foreach (string tag in tags)
                {
                    tagsBuilder.AddItem(tag);
                }
            });
    });

The context parameter is passed by in reference, so there is no copying or boxing. Every delegate is static, so there are no closure allocations.

Generated types and the builder

Everything in this post uses JsonElement and JsonElement.Mutable, but the same builder API works for all generated types. If you have a Person type generated from a JSON Schema, you can create a builder, mutate it, and freeze it in exactly the same way.

The difference is that the generated types only emit .NET members that are compatible with the constraints in their schema. A Person.Mutable will have SetProperty for the properties defined in the schema, and conversions to and from .NET types that match the schema's type constraints. For example, numeric conversions are only available if the schema allows the value to be a number. This means the compiler catches type errors at build time rather than at runtime.

At a glance

Next up

In the next post, we'll look at the standalone evaluator. It's a lightweight code generation mode that produces just a validator and annotation collector, without the full type hierarchy. It's ideal for schema-driven tooling like form generators and configuration editors.

Now let's talk about the foundation that makes V5's performance possible: pooled-memory parsing.

The allocation problem

If you've profiled a .NET service that processes a lot of JSON, you've probably seen a familiar pattern in the GC profiler: a steady stream of small allocations from JsonNode, JsonObject, and JsonArray. Each node in the mutable document model is a separate heap object. For a document with 100 properties, that's 100+ allocations. Each one contributes to GC pressure.

System.Text.Json.JsonDocument solves this with a pooled model, but it's read-only. The moment you need to modify the JSON, you're back to JsonNode and its per-node allocations.

V5's ParsedJsonDocument<T> gives you the best of both: a pooled, read-only document that uses ArrayPool<byte> for all its backing memory, with just 136 bytes of GC pressure per document. This is true regardless of size.

And when you do need to modify things, the JsonDocumentBuilder we'll discuss in the next post uses the same pooled memory model.

Basic usage

Parsing from a string

using var doc = ParsedJsonDocument<JsonElement>.Parse(
    """
    {
        "name": "Alice",
        "age": 30,
        "address": {
            "city": "London",
            "country": "UK"
        }
    }
    """);

JsonElement root = doc.RootElement;
string name = root.GetProperty("name"u8).GetString();
int age = root.GetProperty("age"u8).GetInt32();

The using statement is important. When the document is disposed, all rented memory is returned to ArrayPool. This is the core lifetime rule for ParsedJsonDocument<T>. There are no leaked buffers, and no GC pressure beyond the 136 bytes for the document object itself.

Parsing from UTF-8 bytes

If you already have UTF-8 data from a network buffer, a file read, or an HTTP request body, you can parse directly without transcoding:

ReadOnlyMemory<byte> utf8Data = GetUtf8FromNetwork();
using var doc = ParsedJsonDocument<JsonElement>.Parse(utf8Data);

Async stream parsing

For large files or network streams, async parsing avoids blocking:

using FileStream stream = File.OpenRead("large-data.json");
using var doc = await ParsedJsonDocument<JsonElement>.ParseAsync(stream);

UTF-8 property access

One of the subtle performance wins in V5 is that property names are stored and compared as UTF-8 bytes. The "name"u8 syntax gives you a ReadOnlySpan<byte> - no string allocation, no UTF-16 transcoding.

// Fast: UTF-8 comparison, no allocation
string name = root.GetProperty("name"u8).GetString();

// Also works, but transcodes from UTF-16
string name = root.GetProperty("name").GetString();

For properties you access frequently, V5 can optionally build an O(1) property map for repeated lookups. This is an opt-in feature. You enable it when you know you'll be accessing the same properties repeatedly and want to avoid the cost of linear scanning.

Types are views, not containers

This is a key concept, and it's worth exploring in more detail.

In most .NET serialization frameworks, a deserialized object owns its data. A Person class has a string Name field backed by its own heap-allocated string. The data lives in the object.

In V5, that's not what happens. A generated Person struct is just two fields: a reference to its parent IJsonDocument, and an int index into that document's metadata table. That's it. The struct doesn't hold the string "Alice". It holds a pointer to where "Alice" lives in the document's pooled UTF-8 byte buffer.

┌──────────────────┐      ┌─────────────────────────────────────┐
│  Person struct   │      │   ParsedJsonDocument (pooled)       │
│  ┌────────────┐  │      │  ┌─────────────────────────────┐    │
│  │ _parent ───┼──┼─────▶│  │ MetadataDb (token offsets)  │    │
│  │ _idx: 0    │  │      │  ├─────────────────────────────┤    │
│  └────────────┘  │      │  │ UTF-8 value buffer          │    │
└──────────────────┘      │  │ {"name":"Alice","age":30}   │    │
                          │  └─────────────────────────────┘    │
                          └─────────────────────────────────────┘

This means:

1. Creating a typed view is free. doc.RootElement doesn't copy anything. It returns a struct with the document reference and index 0. Accessing person.Name returns another struct pointing at the same document with a different index.
2. Multiple views share the same data. You can have Person, JsonElement, and Address structs all pointing into the same document. No duplication.
3. The document owns the lifetime. When you dispose the ParsedJsonDocument, the pooled memory goes back to ArrayPool. Any struct that still references it becomes invalid. This is why the using statement matters.

So far we've used JsonElement, but the real power comes from typed documents. Given a generated Person type (from the source generator in post 2):

using var doc = ParsedJsonDocument<Person>.Parse(
    """{"name":"Alice","age":30}""");
Person person = doc.RootElement;

string name = (string)person.Name;      // "Alice"
int age = (int)person.Age;              // 30
bool valid = person.EvaluateSchema();   // true

person is a view. person.Name is a view. Neither allocates. The only allocation is the 136-byte document object itself.

Extended types

V5 supports types beyond what System.Text.Json offers natively:

BigNumber is rarely needed, but when you do need it, it is essential. It's a custom arbitrary-precision decimal type that operates directly on the UTF-8 bytes of the JSON, without intermediate conversion to double or decimal. There is no precision loss and no floating-point surprises.

String and UTF-8 formatting

Every JSON type - JsonElement and all generated types - implements IFormattable, ISpanFormattable, and IUtf8SpanFormattable on .NET 9+. On netstandard2.0 the interfaces aren't available, but the underlying static formatting methods are still there, so you can call them directly. For numeric elements, this means you can format values with standard .NET format strings:

using var doc = ParsedJsonDocument<JsonElement>.Parse("""{"price": 1234.5}""");
JsonElement price = doc.RootElement.GetProperty("price"u8);

// String formatting with culture support
string display = price.ToString("C", CultureInfo.GetCultureInfo("en-GB"));
// "£1,234.50"

// String interpolation (uses IFormattable)
string message = $"Total: {price:N2}";
// "Total: 1,234.50"

For zero-allocation hot paths, write directly to a UTF-8 byte span:

Span<byte> buffer = stackalloc byte[64];
if (price.TryFormat(buffer, out int bytesWritten, "F2", CultureInfo.InvariantCulture))
{
    ReadOnlySpan<byte> utf8Price = buffer.Slice(0, bytesWritten);
    // Write to a Utf8JsonWriter, HTTP response, or log sink - no string allocation
}

The standard numeric format specifiers are all supported: G (general), F (fixed-point), N (number with grouping), E (scientific), C (currency), and P (percentage).

At a glance

Next up

We've seen how V5 pools memory for read-only documents. But what about mutation? In the next post, we'll look at JsonDocumentBuilder and JsonWorkspace. They provide the pooled, version-tracked builder pattern that's at the heart of V5's design trade-off with V4.

Now let's look at what happens when you call person.EvaluateSchema().

Validation is not optional

If you're working with JSON that crosses a trust boundary - API requests, message queues, configuration files, user-supplied data - you need to validate it. Not "check a couple of fields" validate. Schema validate. Against a specification that defines what valid means, and that both producer and consumer agree on.

JSON Schema is that specification, and it covers all the major drafts: 4, 6, 7, 2019-09, and 2020-12 - plus OpenAPI 3.0 and 3.1.

V5 validates over 10× faster than other .NET JSON Schema validators.

The generated validator

If you've followed the source generator approach from the previous post, validation is already built in:

using var doc = ParsedJsonDocument<Person>.Parse(
    """{"name":"Alice","age":30}""");
Person person = doc.RootElement;

bool valid = person.EvaluateSchema();  // true

That's it. The source generator produced a validation method tailored to your specific schema, with every keyword compiled into type-specific checks. There is no interpretation overhead and no runtime schema walking.

Getting diagnostic output

When validation fails and you need to know why, pass a results collector:

using var collector = JsonSchemaResultsCollector.Create(
    JsonSchemaResultsLevel.Detailed);

bool valid = person.EvaluateSchema(collector);

if (!valid)
{
    foreach (var result in collector.EnumerateResults())
    {
        if (!result.IsMatch)
        {
            Console.WriteLine(
                $"  Failed: {result.GetMessageText()} " +
                $"at {result.GetDocumentEvaluationLocationText()}");
        }
    }
}

When you call EvaluateSchema() without a collector, you get a simple pass/fail bool with maximum performance. If you need diagnostics, create a JsonSchemaResultsCollector and choose how much detail you want:

Use the no-collector overload in production hot paths; use Detailed or Verbose for diagnostics.

Dynamic validation at runtime

Not every scenario has schemas known at compile time. Schema registries, configuration validators, and user-supplied schemas all need runtime compilation.

The Corvus.Text.Json.Validator package handles this. Internally, it uses the same code generation engine as the source generator, but invokes Roslyn dynamically:

using Corvus.Text.Json.Validator;

// Load a schema - from a file, string, stream, or URI
JsonSchema schema = JsonSchema.FromFile("Schemas/person.json");

// Validate
bool valid = schema.Validate("""{"name": "Alice", "age": 30}""");

Compiled schemas are cached automatically by their canonical URI, so the first validation incurs a compilation cost (around 100ms) but subsequent calls are sub-millisecond.

Multiple input formats

The Validate() method accepts JSON in whatever format you already have it:

// UTF-8 bytes (zero copy from a network buffer)
bool valid = schema.Validate(utf8Bytes);

// Stream (HTTP request body, file stream)
bool valid = schema.Validate(stream);

// Pre-parsed element
using var doc = ParsedJsonDocument<JsonElement>.Parse(json);
bool valid = schema.Validate(in doc.RootElement);

Pre-loading referenced schemas

If your schema uses $ref to reference other schema files, you can pre-load them to avoid file system or network resolution:

var options = new JsonSchema.Options(
    additionalSchemaFiles: new[]
    {
        new AdditionalSchemaFile(
            "https://example.com/schemas/address.json",
            "Schemas/address.json")
    });

JsonSchema schema = JsonSchema.FromFile("person.json", options: options);

This is particularly useful in CI/CD environments where external resolution may be unreliable or disallowed.

Conformance

We take correctness seriously. The V5 engine passes the official JSON Schema Test Suite for all supported drafts, and we publish results to Bowtie - the cross-implementation conformance project.

Next up

We've seen validation from the caller's perspective. In the next post, we'll go deeper into the V5 memory model. We'll look at how ParsedJsonDocument<T> uses ArrayPool to achieve just 136 bytes per document.

In this post, we'll look at the heart of the developer experience: source-generated types from JSON Schema.

The idea: schema first, types for free

If you've been following our JSON Schema Patterns in .NET series, you'll know we're fans of a schema-first approach. You define your JSON document structure in a schema, and let tooling generate the C# for you.

With V5, that workflow runs at build time with a Roslyn incremental source generator. You annotate a partial struct, point it at a schema file, and get a complete implementation: properties, validation, serialization, implicit conversions, and mutable builders, all with full IntelliSense as you type.

There is no runtime reflection and no code-first attributes. You just need a schema and a struct.

Quick start

1. Create a JSON Schema

Here's a simple schema describing a person:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "required": ["name"],
    "properties": {
        "name": { "type": "string", "minLength": 1 },
        "age": { "type": "integer", "format": "int32", "minimum": 0 }
    }
}

2. Add the NuGet packages

<PackageReference Include="Corvus.Text.Json.SourceGenerator" Version="5.1.0">
  <PrivateAssets>all</PrivateAssets>
  <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
<PackageReference Include="Corvus.Text.Json" Version="5.1.0" />

3. Register the schema and declare a struct

In your .csproj:

<ItemGroup>
  <AdditionalFiles Include="Schemas/person.json" />
</ItemGroup>

In a .cs file:

using Corvus.Text.Json;

namespace MyApp.Models;

[JsonSchemaTypeGenerator("Schemas/person.json")]
public readonly partial struct Person;

4. Use it

Build, and everything is available immediately:

using var doc = ParsedJsonDocument<Person>.Parse(
    """{"name":"Alice","age":30}""");
Person person = doc.RootElement;

string name = (string)person.Name;      // "Alice"
int age = (int)person.Age;              // 30
bool valid = person.EvaluateSchema();   // true

What gets generated

For each [JsonSchemaTypeGenerator] attribute, the source generator produces:

1. Type-safe property accessors for every property in the schema - person.Name, person.Age, etc.
2. Validation via EvaluateSchema() with full support for whatever schema draft your $schema keyword indicates
3. Parsing from strings, byte arrays, streams, and sequences via ParsedJsonDocument<T>.Parse()
4. Serialization via WriteTo(Utf8JsonWriter) and ToString()
5. Implicit conversions to and from .NET primitive types
6. A mutable builder via CreateBuilder(JsonWorkspace) for in-place modification (we'll cover this in a later post)
7. Pattern matching via Match() for oneOf/anyOf discriminated unions
8. Equality operators and GetHashCode()

All generated types are readonly struct values. They're lightweight indexes into pooled JSON data, not heap-allocated objects. This is fundamental to the V5 memory model, and we'll explore it in detail in the post on pooled-memory parsing.

Duck typing: if it validates, it fits

Here's something that might surprise you if you're used to C#'s nominal type system.

Because every generated type is just a view into the same underlying JSON data - a struct containing a document reference and an integer index - you can reinterpret any value as any other type. The conversion doesn't copy data or change the JSON. It just creates a new view with different typed accessors.

// "address" is some JSON element - maybe it came from a JsonElement, maybe
// from a different schema type, maybe from a query result
Address address = Address.From(ref someElement);

// Did it actually validate as an Address?
if (address.EvaluateSchema())
{
    // Yes - we can safely use the typed accessors
    string city = (string)address.City;
}

This is duck typing in the JSON Schema sense: if the JSON data satisfies the schema, you can use it as that type - regardless of where it came from or what type it was originally parsed as.

This is possible because the types don't own or transform the data. They're views. Address.From(ref someElement) doesn't deserialize into an Address. It wraps the same document location in a struct that has City, Street, and Postcode accessors. The data hasn't moved. We've just put on a different pair of glasses.

Pattern matching with Match()

This duck-typing model is the foundation for our oneOf and anyOf support. When your schema says a value can be one of several types, Match() tries each variant's schema in turn and dispatches to the first one that validates:

// Schema: Shape = oneOf [Circle, Rectangle, Triangle]
string description = shape.Match(
    static (in Circle c) => $"Circle with radius {(double)c.Radius}",
    static (in Rectangle r) => $"Rectangle {(double)r.Width}×{(double)r.Height}",
    static (in Triangle t) => $"Triangle with {(int)t.Sides} sides",
    static (in Shape s) => "Unknown shape");

Each branch receives a view of the same JSON data, reinterpreted as the matching type. The lambda parameters are static to avoid accidental closure allocations. There's also a Match overload that accepts a context parameter if you need to pass state in without allocating.

This isn't just syntactic sugar. It's a direct expression of how JSON Schema composition works: oneOf means "exactly one of these schemas validates", and Match() finds which one.

Multi-schema projects

If your schema uses $ref to reference other schema files, you need to register all of them:

<ItemGroup>
  <AdditionalFiles Include="Schemas/**/*.json" />
</ItemGroup>

And you can generate types from multiple definitions within a single schema:

[JsonSchemaTypeGenerator("Schemas/api.json#/$defs/Address")]
public readonly partial struct Address;

[JsonSchemaTypeGenerator("Schemas/api.json#/$defs/PhoneNumber")]
public readonly partial struct PhoneNumber;

The #/$defs/Address part is a JSON Pointer fragment that targets a specific definition within the schema.

Tuning the generator

Several MSBuild properties let you control how types are generated:

<PropertyGroup>
  <!-- Use an explicit cast to string (makes allocations visible) -->
  <CorvusTextJsonUseImplicitOperatorString>false</CorvusTextJsonUseImplicitOperatorString>

  <!-- Treat optional properties as nullable .NET types -->
  <CorvusTextJsonOptionalAsNullable>NullOrUndefined</CorvusTextJsonOptionalAsNullable>

  <!-- Control format keyword enforcement -->
  <CorvusTextJsonAlwaysAssertFormat>true</CorvusTextJsonAlwaysAssertFormat>
</PropertyGroup>

The CLI alternative

The source generator and CLI tool produce identical output. The CLI is useful when you want to pre-generate code, inspect it, or integrate into a pipeline that doesn't use Roslyn:

corvusjson jsonschema Schemas/person.json \
    --rootNamespace MyApp.Models \
    --outputPath Generated/ \
    --outputRootTypeName Person

You can also use --engine V4 to target the V4 code generator instead. It's the same tool and the same schema analysis, but with different output.

Inspecting the generated code

If you want to see what the source generator produces, enable compiler-generated file output:

<PropertyGroup>
  <EmitCompilerGeneratedFiles>true</EmitCompilerGeneratedFiles>
</PropertyGroup>

The generated files will appear under obj/ in your build output. This is handy for debugging or just satisfying curiosity.

Next up

In the next post, we'll look at what happens when you call person.EvaluateSchema(). It's schema validation that runs over 10× faster than other .NET validators.

Today we're introducing something we've been working on for a while: the V5 engine. It is a new code generator and runtime library that sits alongside our existing V4 engine, and represents a fundamentally different set of design trade-offs.

In this series, we'll walk through what V5 brings, why it exists, and how to get started.

The problem we're solving

If you work with JSON in .NET, you've probably noticed a few gaps in the built-in tooling.

System.Text.Json provides excellent low-level readers and writers, and a high-performance serialization framework for code-first projects. But it has no built-in schema validation, no query language support, and its mutable document model (JsonNode) allocates per node with no control over memory lifetime.

Meanwhile, JSON itself has become the lingua franca of APIs, configuration, message passing, and data exchange. You need to parse it, validate it, query it, transform it, patch it, and write it back out. This often happens in a high-throughput request/response pipeline where every allocation matters.

That's what Corvus.Text.Json is for.

Two engines, one toolchain

The V5 engine doesn't replace V4. It offers a different set of trade-offs.

V4 is the right choice when you want the guarantees of an immutable document model - no aliasing bugs, no stale references, and the ability to share values freely across threads.

V5 is the right choice when you're building request/response pipelines, message processing, or other scenarios where you create a document, use it briefly, and dispose it. The builder pattern pools memory across operations, dramatically reducing GC pressure.

Both engines ship in the same CLI tool (corvusjson) and share the same JSON Schema analysis engine. You choose which to use with the --engine flag.

What V5 brings

Pooled-memory parsing

ParsedJsonDocument<T> uses ArrayPool<byte> for its backing memory. It adds just 136 bytes of GC pressure per document, regardless of size, compared with 1,528 bytes for a comparable JsonNode. That's 92% less memory.

using var doc = ParsedJsonDocument<Person>.Parse(
    """{"name":"Alice","age":30}""");
Person person = doc.RootElement;
string name = (string)person.Name;  // "Alice"

Mutable documents with JsonWorkspace

JsonDocumentBuilder<T> and JsonWorkspace provide a pooled builder pattern for creating and modifying JSON in place, with version tracking to catch stale element references at runtime.

using JsonWorkspace workspace = JsonWorkspace.Create();
using var builder = person.CreateBuilder(workspace);
Person.Mutable root = builder.RootElement;
root.SetAge(31);
Console.WriteLine(root.ToString()); // {"name":"Alice","age":31}

Source-generated types

The [JsonSchemaTypeGenerator] attribute triggers a Roslyn incremental source generator that produces strongly-typed C# models from JSON Schema at build time. You get full IntelliSense as you type.

using Corvus.Text.Json;

namespace MyApp.Models;

[JsonSchemaTypeGenerator("Schemas/person.json")]
public readonly partial struct Person;

Schema validation - over 10× faster

Full JSON Schema validation for drafts 4, 6, 7, 2019-09, and 2020-12 (including OpenAPI 3.0 and 3.1). Over 10× faster than other .NET JSON Schema validators, with detailed hierarchical diagnostics when you need them.

bool valid = person.EvaluateSchema();  // true

Three query and transformation languages

V5 includes complete implementations of three JSON query/transformation languages, all with 100% conformance against their official test suites:

• JSONata - a Turing-complete functional query and transformation language. On average 2× faster than Jsonata.Net.Native with 90–100% less memory allocation across 32 benchmarks.
• JMESPath - a standard query language used by AWS CLI and Azure CLI. On average 28× faster than JmesPath.Net with zero allocation across 21 benchmarks.
• JsonLogic - a safe, side-effect-free rule engine for evaluating business rules stored as JSON. On average 3× faster than JsonEverything with zero allocation across 19 benchmarks.

All three support interpreted runtime evaluation, build-time source generation, and CLI code generation.

YAML 1.2 to JSON

YAML is a superset of JSON, and in practice the two are interchangeable in many configuration and data-exchange scenarios. Kubernetes manifests, OpenAPI specifications, GitHub Actions workflows, Azure DevOps pipelines, Helm charts, Docker Compose files - a huge amount of the infrastructure-as-code ecosystem uses YAML as its primary format. If you're processing these in .NET, you eventually need to convert YAML to JSON so you can validate, query, or transform it with standard JSON tooling.

V5 includes a zero-allocation ref struct YAML 1.2 tokenizer that converts directly to a ParsedJsonDocument<T> or a System.Text.Json.JsonDocument with 100% yaml-test-suite conformance. There is no intermediate object model and no per-node allocations.

JSON Patch, JSON Pointer, and extended types

• JSON Patch - RFC 6902 with a fluent PatchBuilder and pooled-memory operations
• JSON Pointer - RFC 6901 zero-allocation path resolution with source location mapping
• BigNumber - arbitrary-precision decimal arithmetic without floating-point conversion
• NodaTime - first-class support for LocalDate, OffsetDateTime, Period, and other temporal types

At a glance: System.Text.Json vs Corvus.Text.Json

Getting started

# Core library
dotnet add package Corvus.Text.Json

# Source generator
dotnet add package Corvus.Text.Json.SourceGenerator

# CLI code generator
dotnet tool install --global Corvus.Json.Cli

What's coming in this series

We'll start with the developer experience: how the source generator and CLI tool turn a JSON Schema into a strongly-typed C# struct with full IntelliSense, and how the standalone evaluator gives you annotation collection for schema-driven tooling like form generators.

Then we'll dig into the runtime. We'll look at how ParsedJsonDocument<T> achieves 136 bytes per document with pooled-memory parsing, how the mutable JsonDocumentBuilder trades V4's immutable safety for in-place mutation with version tracking, and how schema validation runs over 10× faster across all major drafts.

From there, we'll move into the three query and transformation languages. JSONata handles general-purpose JSON transformation, JMESPath provides the concise extraction queries used by AWS and Azure CLIs, and JsonLogic offers safe, side-effect-free business rules you can store in a database and evaluate without code execution.

We'll cover YAML 1.2 conversion with a zero-allocation tokenizer that achieves 100% yaml-test-suite conformance, JSON Patch for RFC 6902 document mutation with a fluent builder, JSON Pointer for zero-allocation path resolution, and V5's extended type system with UTF-8 URI/IRI parsing, arbitrary-precision numerics, and first-class NodaTime integration.

Finally, we'll wrap up with the migration path from V4 to V5, the 10 production Roslyn analyzers that ship with the library, and how to get started.

The answer is compression. VertiPaq uses three encoding techniques, and understanding them is key to understanding why certain model designs perform better than others. Bear with me through the mechanics - it'll pay off when we get to the practical implications in later posts.

Value Encoding

Value encoding is the simplest of the three. The idea is to reduce the number of bits needed per value by applying a mathematical transformation.

A concrete example: say you have a column containing 194, 205, 114, and 216. Storing these raw requires 8 bits per value. But if you instead store values as offsets from 114 (so 80, 91, 0, 102), and the engine knows to add 114 back when reading, you'll need fewer bits. This is a simplified example - In practice, the engine finds the transformation that minimises storage.

Doing this means you can store smaller columns and therefore do faster reads. Again, this adds load to the CPU as the values need to be decoded, but CPU is cheap and getting ever-faster (a recurring theme in this series).

Hash Encoding

Hash encoding is used for any non-numerical values. Instead of storing actual values in the column, VertiPaq stores small integers and keeps a separate dictionary that maps those integers back to the original values.

This is especially powerful when values are repeated. Imagine a column of customer names - large strings appearing across millions of order rows. Without hash encoding, every row stores the full string, which would be a huge amount of memory. With hash encoding, each unique string appears once in the dictionary, and the column itself contains only tiny integers.

Clearly, if a column has very few repeated values (like... for example... an ID column), the dictionary ends up being as large as the column, and you've gained nothing. We'll come back to this when we talk about cardinality in a couple of posts' time.

A couple of details worth knowing:

Any column that participates in a relationship must be hash encoded. This is a VertiPaq requirement for how it handles joins.

Hash encoding also means that VertiPaq is essentially type-independent under the hood. Strings, dates, decimals - everything ends up as hashed integers internally.

Run-Length Encoding

The final technique is run-length encoding (RLE). Instead of storing every value in a column, RLE stores how many times each value repeats in sequence.

For example, a column:

Can instead be stored as (combining run-length and hash encoding):

With a dictionary:

Which is a significant reduction, especially when tables get much larger!

The effectiveness of RLE depends heavily on the sort order of the data. If all the Q1 values were grouped together rather than split across two runs, you'd get even better compression.

The 10-Second Sort Budget

When you import data, VertiPaq analyses the optimal sort order for run-length encoding. By default, it spends 10 seconds on this analysis. If it hasn't found the optimal ordering in that time, it uses the best one found so far.

This is one of the main reasons that reducing the number of columns in your model is a good idea. Fewer columns means fewer permutations to evaluate, making it more likely the engine finds a good sort order within the time budget.

Putting It Together

These techniques work together, but not all at once. Value encoding and hash encoding are mutually exclusive - numeric columns generally use value encoding, while text and other non-numeric data use hash encoding. The exception is that any column participating in a relationship must be hash encoded, so a numeric foreign key column will use hash encoding rather than value encoding. Run-length encoding is then applied on top of whichever encoding is used, compressing consecutive repeated values.

The result is that VertiPaq can hold impressively large datasets in memory - but how well it compresses depends very much on the characteristics of your data. Look out for the next post on why cardinality matters.

Hello again, and welcome to the Optimising DAX series. In this post we're starting at the very bottom of the stack: how VertiPaq actually stores your data when you import it into Power BI.

I'll be honest - before attending Alberto Ferrari's workshop at SQLBits, I had a fairly vague mental model of "it goes into memory and is fast". Turns out there's quite a lot more to it than that, and understanding the storage layer is really the foundation for everything else in this series.

What is VertiPaq?

VertiPaq is the in-memory database engine that powers Power BI's import mode. When you import data into a Power BI model, VertiPaq is what's holding that data in memory and serving it up when your DAX queries ask for it.

The key characteristic of VertiPaq is that it's a column-oriented data store. This means that data is stored column-by-column, rather than the row-by-row approach you'd find in a traditional SQL database.

Why Column-Oriented?

Imagine you want to calculate a SUM over a single column in a table with 20 columns and a million rows. In a row-oriented store, you'd have to read every row - all 20 columns per row - and then throw away 19 of them. You end up reading the full file but throwing away almost everything.

In a column-oriented store, you just read the one column you need. You can directly access the data without touching anything else, which makes reading an entire column (for, e.g., a SUM) incredibly fast.

The Catch: Cross-Column Operations

Column-oriented storage is brilliant for operations on individual columns, but it does add complexity as soon as you need relationships between columns.

A GROUPBY, for example, is more complicated than you might think. The engine has to read each column separately and build up some kind of index map to work out how the values relate to one another. It's a bit more work than just reading rows in order.

However - CPU is very cheap these days (a recurring theme in this series), so more complexity isn't necessarily an issue in practice.

Where things can genuinely break down is with more and more complex queries spanning multiple columns. At a certain point, the engine may just give up and rebuild the entire table from scratch - which is a big cause of slow queries. We'll explore this properly when we get to data materialisation later in the series.

What's Next

So that's the basic storage model - column-oriented, great for aggregations, less great for complex cross-column work. But we've skipped over something important: how does VertiPaq actually fit all this data into memory? The answer is compression, and that's what we'll cover next. Look out for the next post on encoding techniques!

Hello! This is the first in a series of posts about optimising DAX queries in Power BI. The content is based on a full-day workshop I attended at SQLBits 2026 by Alberto Ferrari of SQLBI. I learnt a huge amount about how the underlying engines that power Power BI work, and I wanted to write it all up partly to share it, and partly because I find that writing things down is the only way I actually retain them!

The series is split roughly into two halves. The first covers model optimisation - understanding how VertiPaq stores and compresses your data, and how the design decisions you make at the model level affect performance before you ever write a line of DAX. We look at three common model designs and compare how they perform:

• Star schema - dimension tables (customers, products, dates) linked to a central fact table. Generally the smallest and fastest.
• Flat table - everything denormalised into a single wide table. Actually compresses reasonably well because of all the repeated values.
• Header/detail - two fact tables (e.g. an Orders table and an OrderDetails table) linked by a primary key like OrderID. Spoiler: linking two large fact tables via their primary key is about the worst thing you can possibly do.

The second half covers query execution - how the two engines under the hood (the formula engine and the storage engine) divide up the work, what "data materialisation" means, and practical techniques for keeping your queries in the fast lane.

One thing Alberto stressed at the end of the workshop (and I think it's worth saying up front): at a lot of model sizes, these optimisations aren't worth the effort. Unoptimised DAX is often perfectly fine. These techniques become useful when you're working with larger datasets, hitting real performance problems, or operating under capacity constraints. That said, I do think understanding what's going on under the hood makes you write better DAX even when you're not actively optimising - so hopefully this series is useful regardless!

If you're looking to brush up on DAX fundamentals before diving in, we have plenty of content on the blog. My Learning DAX series covers the basics from filter contexts through to CALCULATE, and Elisenda Gascon's Evaluation Contexts in DAX series is another great starting point. Jessica Hill has written in depth about CALCULATE, context transition, and measures vs calculated columns. Her Performance Optimisation Tools for Power BI post is a particularly good companion to this series.

This post covers what variable libraries are, where you can use them, and some best practices that came out of the session (along with some thoughts of my own).

What Are Variable Libraries?

Variable libraries are Microsoft Fabric's intended mechanism for managing environment-specific configuration. If you've worked with configuration management in software development, this will feel familiar - think appsettings per environment, but for Fabric items.

The basic concept is:

1. You define a set of variables with default values
2. You then create a variable set for each environment (e.g. Dev, Test, Prod)
3. Each variable set needs to be "activated" for the environment it's deployed to

The variable values are all stored in Git, which means they're updated as part of your deployment process. However - and this is important - which variable set is activated for each environment is a separate concern. If you're using Fabric deployment pipelines, you can configure a different active value set per stage, which helps. But it's still worth being aware that activation and deployment are distinct steps, and getting them out of sync is a real risk.

Where Can You Use Them?

Variable libraries can be used across a range of Fabric item types:

• Lakehouse shortcuts - allowing your shortcuts to point to different storage locations per environment
• Notebooks - either directly with %%configure (including setting the default lakehouse) or via notebookutils
• Data pipelines - for parameterising pipeline activities
• Dataflow Gen2 - support has been expanding here, with integration via Variable.Value and Variable.ValueOrDefault functions in M code
• Copy jobs - as source or destination
• User Defined Functions

It's worth noting that the variable values themselves are just stored as part of the Git-tracked configuration, so they can't contain secrets or sensitive information. For secrets, you'd still need to use a solution like Azure Key Vault.

A Best Practice: Don't Map Defaults to Real Environments

One of the most interesting takeaways from this session was around how you structure your variable sets. The presenter's setup had four sets: Default, Dev, Test, and Prod - where the Default set had clearly non-functional placeholder values.

This is in contrast to a pattern where you might map Default directly to your Dev environment, like: Default (= Dev), Test, Prod.

The problem with the latter approach is subtle but significant. If you deploy to an environment and the correct variable set hasn't been activated (or if activation is accidentally skipped), your Fabric items will fall back to the Default values. If Default points to Dev, you could accidentally end up reading from or writing to your Dev environment from a workspace that's supposed to be pointing elsewhere.

However, if your Default values are clearly non-functional placeholders (e.g. a connection string pointing to a non-existent lakehouse), then a missed activation would result in an obvious failure rather than a silent cross-environment issue. Failing loudly is almost always better than failing silently, especially when data integrity is at stake.

Of course, the ideal solution is to properly isolate your workspaces so that cross-environment writes aren't possible in the first place. But adopting the "non-functional default" pattern as an additional safety net is a simple and effective precaution.

Things to Watch Out For

There are a few additional things to be aware of:

SharePoint shortcuts can't currently use variable libraries. This is a notable limitation if you're working with SharePoint data across multiple environments. It's likely coming in a future update, but for now, it means you may need an alternative approach for environment-specific SharePoint integration.

Activation needs attention. While deployment pipelines do support configuring a different active value set per stage, the activation step is still separate from the deployment itself. Building in validation steps (e.g. checking which variable set is active as part of your deployment verification) would help mitigate the risk of things getting out of sync.

Git storage means no secrets. As mentioned, variable values are stored in Git, so anything sensitive needs to be managed separately. Combining variable libraries with Azure Key Vault is a good pattern for handling this.

Summary

Variable libraries are a welcome addition to the Fabric CI/CD story, providing a structured way to manage environment-specific configuration across a range of Fabric items. The key things to remember are: use non-functional default values as a safety net, be aware that activation is a manual step, and combine with Key Vault for any secrets.

I'll be keeping an eye on how this feature evolves - particularly around automation of the activation step and expanded support for item types like SharePoint shortcuts.

This post walks through all five, with practical guidance on when to reach for each one. It's based on a great session by Laura Graham-Brown at SQLBits 2026 - and let's be honest, SharePoint isn't going anywhere in most organisations, so knowing how to get that data into Fabric efficiently is worth doing well.

The Options

Pretty much all of the ingestion options below can work with either a SharePoint list or a folder (which sometimes actually means an entire document library). Let's walk through each one.

1. Dataflow Gen2

Dataflow Gen2 provides a visual, low-code way to ingest and transform data from SharePoint. The demo in the session worked smoothly, and it's a familiar experience if you've used Power Query before.

One important thing to be aware of: Dataflows eat capacity. They're convenient, but if you're running a lot of them, the capacity consumption can add up. Keep this in mind if you're cost-conscious!

An interesting aside: you can actually use Dataflows to ingest from SharePoint on-premises (which is, apparently, still very much a thing in some organisations). This might be the easiest path if you're dealing with legacy on-prem SharePoint installations.

2. Copy Jobs

Copy Jobs are a relatively simple way to move data from SharePoint into Fabric. When you select a list copy, you can set up incremental ingestion - this uses the modified column to only update records that have changed, performing a merge on ID, as described in the Microsoft Fabric Copy Job documentation.

One thing to watch out for: the default schedule for Copy Jobs is every 15 minutes. Depending on your use case, this might be more frequent (and more expensive) than you need. Make sure to review and adjust the schedule to match your actual requirements.

3. Shortcuts

You can create a shortcut directly to a SharePoint folder or document library. This makes the data available in your Lakehouse without physically copying it - the data stays in SharePoint and is referenced in place.

However, there's a significant limitation right now: SharePoint shortcuts can't use variable libraries. This means that if you're working across multiple environments (Dev, Test, Prod), you can't use variable libraries to point the shortcut to different SharePoint locations per environment. The current SharePoint shortcut capabilities and constraints are covered in the Microsoft Fabric OneLake shortcuts documentation.

4. Pipelines

Using pipelines, you can set up a copy activity to ingest SharePoint data. A neat trick highlighted in the session: if you reference a SharePoint list as a folder, it actually lists the files in that folder including the last modified date. This means you can build logic to re-ingest only files that have changed - essentially implementing your own incremental pattern.

5. Notebooks

Notebooks offer the most flexibility, with two main approaches:

Using the Microsoft Graph API - this is a code-first approach where you call the Microsoft Graph API to access SharePoint data programmatically. If you've done something similar in Azure Synapse before, the pattern will be familiar.

Using shortcuts via the Lakehouse - you can add a shortcut to a SharePoint folder or document library, and then read the data directly from the Lakehouse within your notebook. This combines the simplicity of shortcuts with the processing power of Spark.

The notebook approach is the most flexible, but also requires the most technical skill. It's a good fit when you need to do something more complex that the other options don't easily support.

When to Use What?

Here's a rough guide to choosing between the options:

• If you want a quick, visual setup and don't mind the capacity cost - Dataflow Gen2
• If you want simple, scheduled ingestion with incremental support - Copy Jobs (but review that default 15-minute schedule!)
• If you want zero-copy access and aren't working across multiple environments - Shortcuts
• If you already use pipelines - Pipelines with copy activities
• If you need maximum flexibility or are doing complex transformations - Notebooks

Summary

There are more options than you might expect for getting SharePoint data into Fabric, and each has its place. The right choice depends on your specific requirements around cost, complexity, change detection, and multi-environment support.

The biggest gap right now is the lack of variable library support for SharePoint shortcuts, which limits their usefulness in multi-environment setups. But given the pace of Fabric development, I'd expect this to be addressed soon.

As always, the practical reality of working with SharePoint is a bit messy - but at least we now have a decent range of tools to tame the mess!

Across four days, I went to sessions on everything from the SQL Server roadmap and DAX deep dives, to ingesting SharePoint data into Fabric, home automation, inclusive team building, and avoiding burnout. It was a packed few days, and I came away with a lot of new knowledge and plenty to think about.

This post gives an overview of the sessions I attended. I'll also be writing some deeper technical posts on the topics that I found most interesting!

Day 1

Keynote - The SQL Roadmap

Priya Sathy, Shiva Gurumurthy, Bob Ward, Anna Hoffman

The conference kicked off with a Keynote focused on the SQL Server roadmap. There were a lot of announcements about migrating to Azure SQL Managed Instance, including cost estimation tools, the ability to check the effect of changes via lineage, and the fact that you can now manage memory and performance independently. They also touched on VMs - updating to the new series can provide huge throughput increases for analytics, even with smaller VMs.

Anna Hoffman did a couple of great demos. The first used GitHub Copilot to analyse why a stored procedure was running slowly. The second showcased the notebook experience in the mssql VS Code extension, where you can create reports and export them for sharing.

There was also a section on SQL Database in Fabric, highlighting how it's natively integrated into OneLake, autonomous, secure, and optimised for AI. Anna demonstrated using the mssql extension in VS Code whilst connected to Fabric to make changes to a database, and then showing how those changes were reflected in the Fabric UI and could be committed to Git. It's clear that the developer experience for SQL in Fabric is getting a lot of attention.

Finally, there was a section on monitoring in SQL and database agents (all still in preview). This included estate-level triage, asking Copilot for insights, and agents putting alerts straight into Teams and tagging relevant people - a promising step towards more proactive database management.

Variable Libraries in Fabric

Kevin Arnold

Next up was a session on Microsoft Fabric variable libraries - originally scheduled as a 20-minute slot, but they let him continue into the break and he ended up speaking for around 45 minutes (which was a good thing, as there was plenty to cover!).

Variable libraries are the mechanism intended for managing environment-specific configuration in Fabric. The session covered what they are, how they work across different Fabric item types, and some best practices for using them safely.

Watch out for a new post on best practices when using variable libraries!

Microsoft Fabric and the Mess of SharePoint

Laura Graham-Brown

Let's be honest - there's no escaping SharePoint. This session from Laura Graham-Brown explored the various options for ingesting SharePoint data into Microsoft Fabric. She was a great presenter and covered a lot of ground, walking through the practical trade-offs of each approach.

Again, keep an eye out for an upcoming post which will go through this in more detail.

Conversational Analytics at Lloyd's Banking Group

Andrew Herman, Sean Hughes

This was an interesting session about Lloyd's Banking Group's journey from traditional reporting to "GenBI" (Generative BI). The scale involved was impressive - over 100,000 reports, 3 million interactions per month, 27,000 workspaces, and 65,000 active users.

As part of their journey, they did a PoC to prove whether switching from traditional reports to conversational AI would help with the classic challenges of slow insights, inefficiency, and data silos.

What I found most interesting was less the AI angle itself, and more the broader transformation approach they described: proving value early, measuring impact quantitatively, winning senior sponsorship, building capability across the organisation, establishing a single source of truth for key datasets, and baking in CI/CD and governance from the start.

These are solid data transformation principles regardless of whether AI is involved - though the AI angle certainly helps with getting that initial buy-in and excitement!

Fabric Admin Panel

I attended a Fabric admin panel session hoping for discussion around workspace management and change management best practices. It ended up being more of a Q&A driven by the audience, which was mostly made up of Fabric administrators.

A few interesting things came up:

• The Fabric team are considering workspace-level tags (high/medium/low priority), where jobs can then be prioritised accordingly.
• They also recommended building your own Fabric capacity metrics app - which, reading between the lines, suggests the built-in monitoring story still has room for improvement.
• They heavily implied that per-operation metrics are coming, which would be a welcome addition!

There was also an interesting discussion about governance versus usability. The general advice was to lean towards openness and invest in education, with the reasoning being that overly restrictive policies tend to backfire - users will find creative workarounds. (The example given was users screenshotting Power BI reports and extracting data from the screenshots after exports were disabled!)

Shine Bright Like a Star, Without Burning Out!

Gloria Georgieva Clare

As I always try to at conferences, I made sure to attend at least one less technical talk. Gloria Georgieva Clare gave an open and personable session on avoiding burnout, and I'm really glad I went.

She talked about the concept of having four "hobs" - health, social, work, and hobbies - and how you can't have all four firing at full power at any one time. She recommended a few books: The Burnout Society, Fair Play (about unseen, unpaid domestic labour), and Solve Your Stress Cycle.

One of the things that resonated with me most was the idea of defining personal "drivers and guardrails" - concrete commitments like "I want to meet a friend after work once a week" or "I shouldn't be eating lunch at my desk". She also made the important point that just because a stressful situation has ended, doesn't mean the effects of it instantly go away.

I think reflecting on these things semi-regularly is really valuable, and a conference is a good prompt for doing it.

Day 2 - Optimising DAX Queries (Full-Day Workshop)

(Alberto Ferrari - SQLBI)

Day 2 I attended a full-day training workshop with Alberto Ferrari on optimising DAX queries. I learnt a huge amount about how the underlying engines that power Power BI work, and how the in-memory database (VertiPaq) is optimised. We went through real examples of slow queries and why they were slow - often for reasons I would never have known about beforehand.

The workshop covered everything from how VertiPaq stores data using column-oriented storage and various encoding techniques, through to how the formula engine and storage engine work together when executing DAX queries, and practical techniques for identifying and fixing performance bottlenecks.

Watch out for an upcoming series, which will run through this in much more detail.

Day 3

Day 3 was a bit more relaxed. One of the nice things about SQLBits is that the conference social events (in this case, a pub quiz the night before) are actually a good way to meet people. I spent the morning chatting to various people I'd met, which is probably as valuable as any session.

What an Automaton Nerd Learnt by Automating Their Home

Rob Sewell

This was a fun one. Rob Sewell talked about how he'd automated his house using Home Assistant - an open-source home automation platform. He had a network of hundreds of devices all connected via a local network, which was pretty impressive.

What I liked most about this talk was his point that even when you're doing something for fun at home, proper engineering practices and requirements gathering still matter. He had some great examples of what happens when you skip them - including a security setup designed to alert them if someone was in the garden after dark (via a foghorn noise and all the lights turning on), which worked perfectly until they realised they'd forgotten to account for the cat.

People first: building strong, inclusive data teams that actually deliver

Hollie Whittles

This was a 20-minute session, and while it was fairly high-level, there were some points that stuck with me.

Hollie Whittles gave some statistics around how communication is almost always what causes projects to fail, and talked about steps they'd taken to improve it within their team - including anonymous feedback mechanisms and surveys to better understand working styles.

There was a section on neurodiversity that I found particularly interesting. An estimated 30% of people in tech are neurodiverse, but only 16% say there's ever been a conversation about it in their workplace. She highlighted some common workplace practices that can unintentionally disadvantage neurodiverse team members:

Meetings without agendas - people who need more prep time are disadvantaged before the meeting has even begun. Only giving feedback in group settings - many people find public criticism destabilising and need privacy to process constructively. "Open door" policies - which only work for people who are already comfortable stepping through the door.

She also talked about recognising how different team members respond when pressure rises - some withdraw and go quiet, some escalate everything to urgent, and some people-please and agree to everything until they burn out. Being able to recognise these patterns in your team is a useful skill.

Get Creative with Power BI - Making Core Visuals Shine

Valerie Junk

This session was about visualisation and report design in Power BI. A lot of it was fairly straightforward, but there were a few practical tips I picked up.

On tables specifically: use data bars (but not for every column), only highlight what's really important rather than colour-coding everything (just the top and bottom 3, perhaps - I'm definitely guilty of over-colouring), and grey out less important information. Though it's worth noting that heavy conditional formatting can be terrible for accessibility, so having an option to toggle it off is a good idea.

On that note, she showed a neat technique for toggling conditional formatting on and off using a button slicer, without resorting to bookmarks (which, in my experience, can be a bit of a nightmare). The approach is:

1. Create a table with two values: "Formatting" and "No Formatting"
2. Add a button slicer connected to that table
3. Update your conditional formatting rules to check SELECTEDVALUE of that column

Simple, but effective - and avoids the fragility that comes with bookmark-based approaches.

Day 4

Day 4 was the final day, and featured two talks from my colleague Barry Smart.

Spark Unplugged: How In-Process Analytics Is Making Distributed Computing An Expensive Investment

(Barry Smart - endjin)

Barry's first talk challenged the assumption that distributed computing (Spark, in particular) is always the right tool for analytical workloads. With in-process analytics engines like DuckDB and Polars becoming increasingly capable, there's a growing argument that for many workloads, you can get comparable (or better) performance without the overhead and cost of a distributed compute cluster. If you're interested in this topic, keep an eye out for a follow-up post!

No-Compromise Data Apps: Why Streamlit is the Missing Piece in Your Analytics Stack

(Barry Smart - endjin)

Barry's second talk focused on Streamlit - a Python framework for building interactive data applications. The session made the case that Streamlit fills an important gap in the analytics stack: the space between a Jupyter notebook (great for exploration, not great for sharing) and a full web application (powerful, but heavy to build). If you're a data team looking to get interactive tools into the hands of business users without a full frontend development effort, it's well worth a look!

Overall

SQLBits is a well-run conference with a good mix of deep technical content, practical sessions, and softer topics around wellbeing, inclusion, and career development. I came away with a lot of things to think about and try out - which is about all you can ask from a conference, really.

If you're working in the data space - whether that's SQL Server, Fabric, Power BI, or anything in between - it's well worth considering. The 2026 Cartoon theme was a nice touch too!

The Performance Challenge

As part of a recent project, we were building an analytical web API that serves sales data: products, retailers, historical figures and projections, to a React front-end. The data is produced by an ETL process that runs in Databricks and writes the results to Delta tables in a data lake. The natural way to query that data is through Databricks SQL Serverless: it handles the complex analytical workloads, scales well, and integrates cleanly with the rest of the stack.

There's a catch, though. Databricks SQL Serverless clusters can be paused when idle, and cold-start latency can add several seconds—sometimes tens of seconds to the first request after a period of inactivity. Even on a warm cluster, query execution time for some of the larger datasets runs into multiple seconds. For a web API that needs to feel responsive, that's a problem.

The good news is that most of the data retrieved from Databricks is reference data: it changes infrequently, and when it does change, it changes in a controlled, versioned way. That observation is what makes aggressive caching safe, and it's the insight that shaped the approach described here.

Key observation: the application works with specific named versions of the sales data. Within a given version, the data is completely immutable. There's no risk of serving stale data from a cache, because the data simply doesn't change once a version is published.

Understanding the Data Access Requirements

All data access in the application flows through a single interface, ISalesDataRepository. Here's a trimmed version of its key methods:

public interface ISalesDataRepository
{
    Task<string> GetLatestVersionIdAsync();
    Task<VersionInfo> GetVersionAsync(string versionId);
    Task<Product[]> GetProductsAsync(string versionId);
    Task<Retailer[]> GetRetailersAsync(string versionId);
    Task<SalesSummary[]> GetSalesAsync(string versionId, DateRange dateRange);
    Task<SalesSummary[]> GetSalesByIdsAsync(string versionId, string[] ids);
}

The versioning model is central to everything. A version is created by the ETL process and is immutable once published. The application operates within a specific version, most requests include a versionId that scopes the data being retrieved.

Not all data falls into the same category, though. When we analyse the methods, three distinct types emerge:

• Fully immutable within a version — products, retailers, and other reference data. Once fetched for a given version, these can be cached indefinitely. They will never change.
• Near-real-time — the latest version ID. This needs a short time-to-live (we use five minutes) to pick up new versions when they're published without hammering the source on every request.
• On-demand lookups — targeted queries like GetSalesByIdsAsync, where the combination of parameters is effectively unbounded. Caching these isn't meaningful; they always go direct to Databricks.

That analysis drives a selective caching strategy rather than a blanket one. Not everything is worth caching, and not everything can be cached with the same TTL.

The Decorator Pattern: A Primer

If you haven't used the Decorator pattern before, the idea is straightforward. A decorator implements the same interface as the class it wraps. It adds behaviour: before, after, or instead of, delegating calls to the inner implementation. The consumer doesn't know, or care, whether it's talking to a "real" implementation or a decorator. It just uses ISalesDataRepository.

This is a natural fit for caching. The actual data access code stays clean and focused on talking to Databricks. The caching logic lives entirely in the decorators. The two concerns don't touch each other.

In our case, the chain looks like this:

Client
  └─► MemoryCachingSalesDataRepository             (Layer 2: in-process, sub-millisecond)
        └─► BlobStorageCachingSalesDataRepository  (Layer 1: shared, persistent, fast)
              └─► SalesDataRepository              (Real implementation: Databricks SQL)

Each layer wraps the one below it. The MemoryCachingSalesDataRepository doesn't know it's wrapping a blob cache, it just knows it has an ISalesDataRepository to delegate to when it misses. Dependency injection wires the chain together; the decorators themselves have no knowledge of each other.

Layer 1: Azure Blob Storage Cache

Why Blob Storage?

An in-process memory cache is lost when the API restarts or when a new instance spins up. Our application runs in Azure Container Apps, which scales out to multiple replicas and restarts during deployments. Without a persistent cache layer, every new instance would need to hit Databricks on its first request, exactly the cold-start problem we're trying to avoid.

Blob Storage is cheap, fast for reads, and shared across all replicas. It's not as fast as in-process memory, but it's orders of magnitude faster than waiting for a Databricks cluster to warm up.

How It Works

BlobStorageCachingSalesDataRepository implements ISalesDataRepository and wraps the real SalesDataRepository. For each cacheable method, it constructs a deterministic blob path based on the data type, version ID, and any relevant parameters, for example, sales/{versionId}/products.bin for the products list.

The core of the implementation is a generic GetOrCreateAsync<T>() helper:

private async Task<T> GetOrCreateAsync<T>(
    string blobPath,
    Func<Task<T>> factory)
{
    var blobClient = _containerClient.GetBlobClient(blobPath);

    if (await blobClient.ExistsAsync())
    {
        var content = await blobClient.DownloadContentAsync();
        return MemoryPackSerializer.Deserialize<T>(content.Value.Content.ToArray());
    }

    // Cache miss: fetch from inner repository
    var result = await factory();

    // Write to blob storage for next time
    var bytes = MemoryPackSerializer.Serialize(result);
    await blobClient.UploadAsync(BinaryData.FromBytes(bytes), overwrite: true);

    return result;
}

The data is stored as binary blobs using MemoryPack, a high-performance binary serialiser for .NET. Compared to JSON, this keeps blob sizes small and deserialisation fast, both matter at the scale of multiple API instances reading from shared storage.

Preventing the Thundering Herd

A naïve implementation has a race condition that's particularly nasty during cold starts. If the blob doesn't exist and multiple requests arrive concurrently, which is exactly what happens when a new container instance starts up and the front-end fires several API calls at once, they all miss the cache and all hit Databricks simultaneously.

This is the thundering herd problem. In the worst case, you end up making dozens of parallel queries to a cluster that's still warming up.

The solution is a ConcurrentDictionary<string, SemaphoreSlim> keyed by blob path. When a cache miss occurs, we acquire the semaphore for that blob path before proceeding. Inside the lock, we check the blob again, another request may have already populated it while we were waiting. If it's still a miss, we fetch from the inner repository and write the result. Here's the pattern in full:

private async Task<T> GetOrCreateAsync<T>(
    string blobPath,
    Func<Task<T>> factory)
{
    var blobClient = _containerClient.GetBlobClient(blobPath);

    // Fast path: blob already exists
    if (await blobClient.ExistsAsync())
    {
        var content = await blobClient.DownloadContentAsync();
        return MemoryPackSerializer.Deserialize<T>(content.Value.Content.ToArray());
    }

    // Slow path: acquire per-blob semaphore to prevent thundering herd
    var semaphore = _semaphores.GetOrAdd(blobPath, _ => new SemaphoreSlim(1, 1));
    await semaphore.WaitAsync();

    try
    {
        // Double-check: another waiter may have populated the blob
        if (await blobClient.ExistsAsync())
        {
            var content = await blobClient.DownloadContentAsync();
            return MemoryPackSerializer.Deserialize<T>(content.Value.Content.ToArray());
        }

        var result = await factory();

        var bytes = MemoryPackSerializer.Serialize(result);
        await blobClient.UploadAsync(BinaryData.FromBytes(bytes), overwrite: true);

        return result;
    }
    finally
    {
        semaphore.Release();
    }
}

Only one request per unique blob path reaches Databricks. All others wait for the semaphore, benefit from the result, and return immediately. It's worth noting that this applies per instance, if there are multiple container replicas running, each will independently populate its own copy of the blob. In practice that's fine: the first request to any instance pays the Databricks cost; subsequent requests benefit from the cached blob. If it does become a problem, there are more complex solutions using shared locks, for example, Corvus.Leasing which uses Azure blob storage to provide a means to acquire, release and extend exclusive leases to mediate resource access in distributed processing.

What's Deliberately Not Cached Here

Version lookups, GetLatestVersionIdAsync and GetAvailableVersionsAsync always go to the source. We want the application to notice when a new version is published within a reasonable time. Caching these in blob storage would give us no meaningful benefit over the in-memory TTL we apply at the layer above.

Targeted on-demand lookups by ID also bypass the blob cache. The combination space, different sets of IDs against different versions, is too large to cache meaningfully.

Graceful Degradation

All blob I/O is wrapped in try/catch. If the cache layer fails for any reason—transient connectivity, permissions, a bad serialisation, it logs a warning and falls through to the inner repository. The application keeps working; it's just slower until the cache is warm again.

Layer 2: In-Memory Cache

Why a Second Layer?

Even a fast Blob Storage read involves a network round-trip and deserialisation overhead. For a busy API serving the same reference data many times per second, that adds up. IMemoryCache keeps deserialised objects in the process's heap. Reads are effectively instantaneous, no network, no deserialisation, just a dictionary lookup.

How It Works

MemoryCachingSalesDataRepository wraps the Blob Storage decorator (which in turn wraps the real repository). It uses the standard cache.GetOrCreateAsync() pattern, with expiry configured per data type:

• Immutable data within a version: no expiry, held in memory until the process restarts.
• Latest version ID: five-minute sliding expiry, so new versions are picked up in a timely manner.

Cache keys incorporate the version ID where relevant, so different versions don't collide.

Here's a representative method:

public async Task<Product[]> GetProductsAsync(string versionId)
{
    using var activity = _activitySource.StartActivity("GetProducts");

    var cacheKey = $"{ProductsCacheKeyPrefix}{versionId}";

    if (_cache.TryGetValue(cacheKey, out Product[]? cached))
    {
        activity?.SetTag("cache.hit", true);
        return cached!;
    }

    activity?.SetTag("cache.hit", false);

    var result = await _innerRepository.GetProductsAsync(versionId);
    _cache.Set(cacheKey, result);

    return result;
}

Observability with Activity Source

Each method creates an Activity via ActivitySource, which participates in distributed tracing through OpenTelemetry. We record whether the request was a cache hit or miss as a tag on the activity: cache.hit = true/false.

This turns out to be genuinely useful. When we look at the observability dashboard, we can see at a glance what proportion of requests are being served from memory versus falling through to lower layers. It's how we validated that the cache was actually working as expected after deployment.

What Doesn't Get Cached In Memory

Targeted on-demand lookups by ID, as with the blob layer, always delegate to the inner repository. The combination space makes in-memory caching impractical, we'd risk holding enormous amounts of data with a very low hit rate.

Wiring It Together with Dependency Injection

The decorator chain is composed in the service registration. The order matters: each decorator needs to wrap the layer below it, not the one above. We register the concrete types first, then register the ISalesDataRepository abstraction as the fully-composed outermost decorator:

// Innermost: the real Databricks implementation
services.AddSingleton<SalesDataRepository>();

// Middle layer: Blob Storage cache wrapping the real implementation
services.AddSingleton<BlobStorageCachingSalesDataRepository>(sp =>
    new BlobStorageCachingSalesDataRepository(
        sp.GetRequiredService<SalesDataRepository>(),
        sp.GetRequiredService<BlobServiceClient>(),
        sp.GetRequiredService<IOptions<SalesDataBlobCacheOptions>>(),
        sp.GetRequiredService<ILogger<BlobStorageCachingSalesDataRepository>>(),
        sp.GetRequiredService<ActivitySource>()
    )
);

// Outermost: in-memory cache wrapping the blob storage cache
// This is what consumers receive when they depend on ISalesDataRepository
services.AddSingleton<ISalesDataRepository>(sp =>
    new MemoryCachingSalesDataRepository(
        sp.GetRequiredService<BlobStorageCachingSalesDataRepository>(),
        sp.GetRequiredService<IMemoryCache>(),
        sp.GetRequiredService<ActivitySource>()
    )
);

Any component that depends on ISalesDataRepository via the DI container automatically gets the fully-composed chain. The decorators themselves have no knowledge of how they're composed, they just know they have an ISalesDataRepository to delegate to.

Tracing a Request Through the Cache Layers

Let's walk through what happens for a call to GetProductsAsync under three different scenarios.

First Request After Deployment (Cold Everything)

1. MemoryCachingSalesDataRepository — cache miss; delegates to inner.
2. BlobStorageCachingSalesDataRepository — blob not found; acquires semaphore; delegates to inner.
3. SalesDataRepository — queries Databricks SQL. If the cluster has been idle, this may take several seconds while it warms up.
4. The result flows back up: written to Blob Storage, then stored in the in-process cache.

This is the expensive path. It only happens once per unique dataset per application instance.

Second Request in the Same Process

1. MemoryCachingSalesDataRepository — cache hit; returns immediately from memory.

That's it. Sub-millisecond response time regardless of what Databricks is doing.

First Request After a Replica Starts (or a Process Restart)

1. MemoryCachingSalesDataRepository — cache miss (new process, empty memory cache).
2. BlobStorageCachingSalesDataRepository — blob found; deserialises and returns.
3. The result is held in memory for all subsequent requests.

The new instance pays a Blob Storage round-trip on its first request for each dataset, but never needs to hit Databricks. The warm-up time for a new replica is a handful of Blob Storage reads rather than a cluster cold-start.

Results and Trade-offs

What We Gained

After the initial warm-up, the vast majority of read requests are served from the in-memory cache in sub-millisecond time. New replicas warm quickly from Blob Storage without touching Databricks. The thundering herd problem is eliminated: Databricks is hit at most once per unique dataset per version per application instance, regardless of how many concurrent requests arrive.

Honest Trade-offs

It would be misleading to present this as a straightforward win with no downsides. There are real trade-offs:

• Complexity. We now have three classes where one might seem simpler at first glance. Each decorator is individually straightforward, but the chain requires understanding to navigate.
• Staleness by design. The latest version ID has a five-minute lag. The application has to tolerate that, and the product team has to accept it. In our case that's fine, new versions aren't published on a minute-by-minute basis, but it's a deliberate constraint.
• Cache invalidation is implicit. When a new version is published, the old version's blobs remain in Blob Storage, they're just never requested for that version again. A separate cleanup process could remove them if storage cost becomes a concern, but for now the cost is negligible.
• Memory pressure. Keeping large datasets in IMemoryCache indefinitely is a deliberate choice that works because our process's memory budget accommodates it. For larger datasets or more memory-constrained environments, you'd want to think carefully about size limits and eviction policies.
• The Blob Storage layer adds latency on cold reads. If the Databricks cluster happens to be warm when a blob is missing, going via Blob Storage is actually slower than going direct. In practice, the cluster being warm on a cold-start scenario is the exception rather than the rule, but it's worth being aware of.

As always, the answer is "it depends." This approach made sense for our workload profile. For a different set of constraints, larger datasets, more frequent version changes, tighter memory budgets — some of these trade-offs might tip the other way.

Conclusions

The Decorator pattern is a clean fit for layered caching because it keeps caching logic entirely separate from data access logic. Adding a new cache tier is additive, it doesn't require changes to existing classes. The chain is composed by the DI configuration, not by the decorators themselves.

The design decisions that made this work were:

1. Understanding which data is truly immutable. The versioning model gave us a strong guarantee that made aggressive, indefinite caching safe.
2. Choosing the right storage tier for each layer. Blob Storage for persistence and cross-replica sharing; IMemoryCache for the fast path.
3. Protecting against the thundering herd. The semaphore-based double-check at the Blob Storage layer is easy to overlook but critical at cold start.

Databricks SQL Serverless is a powerful analytical query engine. The trick is to use it for what it's good at, processing and transforming large analytical datasets, and let fast caches absorb the high-frequency, low-latency reads that a web API demands. The Decorator pattern gives us the architectural seam to do that cleanly.

The same pattern applies well beyond Databricks. Anywhere you have a slow or expensive data source serving data that changes infrequently, layering caches using decorators is a maintainable and extensible approach worth considering.

If you've got any questions or would like to discuss anything we've talked about, please feel free to leave a comment below.

endjin/fabric-performance-benchmark

The Broader Context: A Platform Responding to a Shift

This benchmarking study doesn't exist in isolation. It is the latest instalment in our Adventures in Polars and DuckDB series, in which we have been making the case that a fundamental shift is underway in how organisations approach analytical data processing.

The argument, which we have explored in depth across both series, centres on what Hannes Mühleisen (co-creator of DuckDB and Professor of Data Engineering at the University of Nijmegen) calls the "data singularity": the point at which the processing power of mainstream single-node machines surpasses the requirements of the vast majority of analytical workloads. We introduced this concept in DuckDB: The Rise of In-Process Analytics and Data Singularity and revisited it from a Polars perspective in Why Polars Matters for Decision Makers.

The core observation is straightforward: CPU core counts, RAM, and NVMe storage throughput have all improved dramatically over the past decade, while the size of most useful analytical datasets has grown far more slowly. Amazon's own internal Redshift telemetry, analysed by MotherDuck, suggests we are already close to the singularity — the 99th percentile of datasets in a production big data platform fits comfortably on a modern laptop. Their data suggests we are spending around 94% of query dollars on computation that doesn't actually need big data infrastructure.

The problem, historically, was that most data tools were designed before this shift. They could not take advantage of modern hardware capabilities. That gap gave rise to a new generation of in-process analytics engines which are built to exploit the full potential of a single, well-resourced machine. Two tools stand out: DuckDB for those who prefer SQL, and Polars for those who prefer a DataFrame API. Both re-engineer the analytics stack from the ground up: column-oriented storage, vectorized execution across all available CPU cores, and intelligent query planning that eliminates unnecessary work. Neither requires a cluster. Neither has network overhead, authentication complexity, or the coordination costs of distributed systems.

The practical implication, and the hypothesis this benchmarking study was designed to test, is that for the majority of enterprise analytical workloads, these tools running on a single well-resourced node will outperform Spark at a fraction of the cost.

Microsoft Fabric's Response: The Python Notebook

It is telling that Microsoft has recognised this shift at a platform level. The introduction of Python Notebooks to Microsoft Fabric is a direct response to the in-process analytics movement.

Where Fabric's Spark Notebooks provision a distributed cluster on demand (with all the associated overhead in spin-up time, coordination cost, and capacity consumption) Python Notebooks provide a single, configurable execution node. They come pre-installed with both DuckDB and Polars, a notable design choice that signals Microsoft's acknowledgement that these tools have earned their place in the enterprise data stack. Microsoft explicitly recommends both as alternatives to Pandas for memory-intensive workloads.

In parallel, both DuckDB and Polars have added direct support for OneLake — the storage platform that underpins Fabric. DuckDB's native Delta extension enables querying Delta tables stored in a Fabric Lakehouse via delta_scan(), reading directly from OneLake paths without additional configuration. Polars similarly supports reading and writing Delta tables via pl.read_delta(), pl.scan_delta(), and df.write_delta(), with OneLake authentication handled through Fabric's notebookutils. Both tools also support standard ABFS paths, enabling direct interaction with raw files in the Lakehouse Files area. We have covered these integration patterns in detail in DuckDB Workloads on Microsoft Fabric and Polars Workloads on Microsoft Fabric.

The Python Notebook is configurable across a range of single-node sizes (2, 4, 8, 16, 32, and 64 vCores, with memory scaling proportionally). This is a meaningful range: a 32-vCore Python Notebook gives DuckDB or Polars access to substantially more parallelism than even a well-resourced Spark executor, without any of the coordination overhead of distributed execution.

What follows is our attempt to put hard numbers behind these claims, using a realistic enterprise workload on real Fabric infrastructure.

Data Source

The use case is implemented using open data provided by the UK Land Registry House Price Data open data repository, made available under an Open Government Licence.

The data is provided as a set of CSV files (one per calendar year) which have been downloaded to a Fabric Lakehouse. The dataset covers property sales since 1995, with approximately 1 million transactions per year. In total, 30 years of historic data amounts to roughly 30 million rows and ~5GB of raw CSV.

This scale is deliberate. We find that many published benchmarks focus on processing datasets that are rarely encountered in practice. Our objective is to focus on the scale we more commonly encounter in enterprise client engagements — medium-scale datasets that are interesting enough to stress-test the differences between engines, but representative of the workloads that most data teams actually run day to day. The data fits in memory for all configurations tested, which is precisely where in-process engines are designed to excel.

Use Case

The use case mimics a common set of data transformations typical for data of this nature:

1. Start Up & Set Up — provisioning the platform (Spark or Python), then completing startup tasks such as importing Python packages.
2. Ingestion & Transform — reading raw data from a set of CSV files, standardising, cleaning, and adding derived features.
3. Write Dimensional Model — writing different slices of the transformed data to the Lakehouse in Delta format for downstream consumption, in this case as a dimensional model for Power BI.
4. Read and Summarise — reading the Delta tables back and running analysis based on filtering, joining, and summarising data across different dimensions.
5. Benchmark Capture & Clean Up — capturing timestamps and memory consumption metrics at each stage and persisting them to the Lakehouse for analysis.

flowchart TB
A["1. Start Up & Set Up"] --> B["2. Ingestion & Transform"]
B --> C["3. Write Dimensional Model"]
C --> D["4. Read & Summarise"]
D --> E["5. Benchmark Capture & Clean Up"]

This pipeline exercises the full range of operations that matter to data engineers: reading raw files at scale, executing complex transformations, writing Delta tables, and querying structured data back. It is a fair test of what each engine is actually optimised to do.

Fabric Platforms

Fabric offers multiple notebook environments. For this study we used two:

• Spark notebooks — a notebook experience over an on-demand Spark cluster, configurable in terms of vCores, memory, and number of executor nodes. Enables polyglot development (Python, R, SQL) across a distributed compute environment.

• Python notebooks — a more recent addition to Fabric. Python notebooks provision a single execution node sized according to pre-defined configurations (vCores and memory). Whilst positioned for "smaller" workloads, we find that the majority of enterprise use cases can be comfortably accommodated on this platform when the right tooling is chosen.

Workloads

The study compares four data processing engines running the same use case on Fabric:

1. Pandas — the default Python library for data engineering. Vast ecosystem, but single-threaded and constrained to datasets that fit in memory. Serves as a baseline.

2. PySpark — the Python API for Apache Spark. Designed for distributed computation across clusters and widely deployed via Databricks and Azure Synapse. The incumbent choice for enterprise-scale data engineering.

3. Polars — a Rust-based engine with a Python API. Designed from the ground up to exploit modern hardware: automatic parallelisation across all available cores, lazy evaluation with query plan optimisation, and memory-efficient columnar processing. We explored Polars' technical foundations in detail in Polars Technical Deep Dive.

4. DuckDB — a C++ in-process analytical database with a Python API. Optimised for OLAP workloads, capable of querying CSV files and Delta tables directly, and able to use disk for larger-than-memory datasets. We covered DuckDB's internals — columnar storage, vectorized execution, zone maps — in DuckDB In Depth: How It Works and What Makes It Fast.

Fabric Capacity and Capacity Units (CUs)

A Fabric capacity is a dedicated pool of compute resources purchased from Azure — a fixed amount of computational horsepower continuously available to workspaces assigned to it. When you purchase a Fabric capacity (e.g. F8, F64), you are reserving that number of capacity units (CUs) for continuous use across all workloads: notebooks, pipelines, warehouses, Power BI, and so on.

CUs are Fabric's abstraction layer for billing compute across heterogeneous workloads. Different engines have different conversion rates, meaning your F64 capacity represents different amounts of practical compute depending on which engine is consuming it.

Consumption is measured in CU Seconds: the number of CUs consumed multiplied by duration in seconds. The fundamental formula for both Spark and Python Notebooks in Fabric is 0.5 CU per second per vCore, though for Spark the total vCore count must account for driver and all executor nodes:

• A Python Notebook sized at 8 vCores consumes 4 CUs per second.
• A Spark Notebook with 1 driver (8 vCores) and 1 executor (8 vCores) consumes 8 CUs per second.

This billing asymmetry is significant and, as the results below will show, it creates a strong economic case for Python Notebooks when the workload doesn't genuinely require distributed execution.

Configurations

Achieving direct parity across Spark and Python notebook platforms is not straightforward. We opted for configurations that allow a range of CU-per-second comparisons, including some like-for-like points:

The default Python Notebook (2 vCores, 16GB RAM) is 8x cheaper per second to run than the default Spark Notebook (1 Executor 8/8 vCores 56G/56G RAM). The smallest Spark configuration (1 Executor 4/4 vCores 28G/28G) is equivalent in CU cost to the 8-vCore Python Notebook which provides ~2.5x the RAM, and as the results show, very significant computational throughput when running DuckDB or Polars.

Spin-up times are also materially different, and matter for development workflows. Python Notebooks at the default 2-vCore configuration provision in under 30 seconds. Spark clusters in this study took between 3 and 3.5 minutes at all configurations tested:

When developing directly in Fabric notebooks we use the default platform configurations operating on test data. Then scale up the configuration as needed in production.

With DuckDB and Polars, we favour local development given the access this gives us to modern IDEs, coding agents and unit testing frameworks. Faster provisioning directly improves the developer inner loop. Shorter iteration cycles during development compound over the course of a project. This is consistent with our experience migrating client workloads from Spark to in-process engines, where test suite runtimes dropped from minutes to seconds and local development became a practical reality again.

Methodology

Multiple runs were completed for each combination of engine, workload, and configuration. Median execution times are reported throughout to reduce the influence of outliers.

Analysis

Elapsed Time Analysis

Elapsed time analysis includes the time to spin up the required Spark or Python environment. Environments with faster provisioning therefore have an inherent advantage here. Note that spin-up time does not incur a CU cost on Fabric — if cost is your primary concern, skip ahead to the Execution Time Analysis.

The table below shows median elapsed times for each workload on its default environment:

Key observations:

• PySpark (on its default 8-CU environment) is the fastest at elapsed time (~126 seconds), but only marginally so.
• DuckDB on the default Python Notebook (1 CU per second) runs in ~134 seconds — just 6% slower — at one-eighth the CU cost.
• Polars on the same minimal configuration completes in ~175 seconds, also at one-eighth the cost.
• Pandas, the incumbent default, takes over four minutes — roughly twice Spark's elapsed time.

Execution Time Analysis

Execution time excludes spin-up and environment provisioning, enabling a direct comparison between engines across configurations at equivalent CU costs.

The Top 10 results are presented below:

• DuckDB achieves the fastest pure execution time, on an 8-vCore Python Notebook with 64GB RAM.
• DuckDB and Polars occupy 8 of the top 10 positions before Spark appears.
• The fastest Spark execution time is more than twice that of DuckDB — which achieves its best result on infrastructure with half the CU cost.

The line chart below shows median execution times across all engines and configurations, with CUs per second as the common measure of both environment size and cost:

Key observations:

• Execution time for all engines initially decreases as more cores and memory become available, but there is a clear "sweet spot" beyond which performance plateaus or even degrades slightly. Throwing more infrastructure at a problem does not guarantee faster results and can actually make things slower.
• At comparable CU levels (e.g. 4 CUs per second), DuckDB and Polars on Python Notebooks significantly outperform PySpark on Spark Notebooks.

This "sweet spot" behaviour is consistent with what we know about how DuckDB and Polars are engineered. Both tools use automatic parallelisation across available cores, but there is an overhead to thread coordination that grows with core count. Beyond the point where all available parallelism is fully utilised, adding more cores yields diminishing returns.

CU Cost Analysis

Shifting focus from execution time to cost, the following table lists the 10 cheapest engine/configuration combinations:

The cheapest Spark run costs more than 5x the cheapest DuckDB run and approximately 4x the cheapest Polars run. Even Pandas on the default minimal Python Notebook is cheaper than the most economical Spark configuration.

Execution Time versus Cost

The following scatter chart provides a two-dimensional view of all engine/configuration combinations tested. The horizontal x-axis shows execution time; the vertical y-axis shows cost on a logarithmic scale.

The sweet spot (best combination of fast execution and low cost) lies in the bottom-left quadrant. That space is dominated by DuckDB and Polars.

Stage Analysis: What Separates DuckDB from Polars?

Both DuckDB and Polars are modern in-process query engines built for exactly this kind of workload, yet DuckDB consistently comes out ahead. The stage-level analysis reveals where that advantage is won.

DuckDB's edge appears consistently in the stages that involve reading from the Fabric Lakehouse — specifically, reading Delta format tables. DuckDB uses its own native Delta extension for this purpose, reading directly from OneLake without additional dependencies. Polars, by contrast, currently uses the delta-rs package for Delta reads.

This is an important distinction in the context of Microsoft Fabric's architecture. OneLake stores data in Delta format, and any engine that can query Delta tables natively has a structural advantage. It is worth watching whether Polars develops its own native Delta reader over time; if it does, the gap between the two engines may narrow.

Conclusions

This benchmarking study provides concrete, Fabric-specific evidence for what we have been arguing across the broader series: in-process analytics engines running on single-node infrastructure are a serious and, for most workloads, superior alternative to distributed Spark.

1. Modern in-process engines outperform Spark for medium-scale workloads — DuckDB and Polars delivered faster execution times than PySpark across all comparable configurations, often by a factor of 2x or more. The claims we made in our DuckDB and Polars series hold up under real Fabric workloads.

2. Cost efficiency strongly favours Python Notebooks with modern engines — the cheapest Spark configuration costs 4-5x more than the cheapest DuckDB run for equivalent work. For teams with finite Fabric capacity budgets, this is a material consideration. Profile your workload, start small, and scale to Spark only when you genuinely need it.

3. More resources don't always mean faster execution — there is a "sweet spot" for resource allocation beyond which performance plateaus or degrades. This challenges the intuition that scaling infrastructure will proportionally improve throughput. Both DuckDB and Polars are efficient enough that 4-8 vCores often delivers the best balance of speed and cost.

4. Default configurations are not equal — and the gap matters — the default Python Notebook (2 vCores) is 8x cheaper per second to run than the default Spark Notebook, yet delivers comparable elapsed-time performance when using DuckDB or Polars. For development workloads, the Python Notebook should be the default choice.

5. DuckDB's native Delta reader provides a measurable edge — the stage analysis suggests DuckDB's advantage over Polars comes primarily from its native Delta reading capability. This is a meaningful finding in the context of Fabric, where Delta is the dominant table format. It reinforces one of DuckDB's core design principles: eliminate friction wherever data meets compute.

6. Microsoft is responding to the in-process analytics movement — the introduction of Python Notebooks pre-installed with DuckDB and Polars, combined with OneLake support in both tools, signals that the platform is evolving to accommodate this shift. Teams investing in these tools are aligned with the direction of travel, not swimming against it.

Note - all code used to generate this benchmark and the supporting analysis is available in a public repo on GitHub.

endjin/fabric-performance-benchmark

The data singularity is not a theoretical future state — it is arriving now, and platforms like Microsoft Fabric are starting to reflect that. For most enterprise analytical workloads, the question is no longer whether single-node in-process engines can handle the job. Based on the evidence here, they can deliver a faster and cheaper solution than the distributed alternative.

So which workloads do still justify Spark's overhead? Reserve Spark for datasets that genuinely exceed single-node memory capacity, or for workloads where Fabric-specific Spark optimisations (V-ORDER, Liquid Clustering) are demonstrably valuable. For everything else, DuckDB and Polars on a Python Notebook are the pragmatic choice.

The good news is that Microsoft Fabric makes all of these compute options available to you, underpinned by Delta format and OneLake as the common storage layer. So you are not forced to make a choice up front, you have the flexibility to adapt without being forced to move data or adopt a different platform.

I've worked in technology for a long time. In fact, I've had a 25-year long career in software and data, which means I've had to use a lot of different programming languages, frameworks, platforms and tech stacks. But it's also meant that I've used Excel a lot. I would describe myself as fairly competent in Excel. Not world championship level by any means, and probably not even Joel Spolsky level (although I did start to suck slightly less after watching this 10 years ago). But fairly competent.

However, I came across an interesting article recently that really grabbed my attention. This post on How-To Geek describes a "three-tab rule" in Excel, separating source data, logic, and presentation. It's framed by making comparisons to the Model-View-Controller (MVC) pattern in web development, something I'm very familiar with, but had never considered applying to Excel. I was pretty surprised that I'd never come across this before (especially as I've spent a lot of time thinking about engineering practices in software), but I was also immediately struck by how useful, sensible and simple it was.

The pattern described proposes structuring Excel workbooks using three distinct tabs: Source, Logic, and Interface. The premise is straightforward - most spreadsheets fail because they mix raw data, calculations, and final reports on the same screen. By separating these concerns, you create workbooks that are easier to audit, maintain, and scale.

The MVC-based pattern works like this: the Source tab holds your raw, unmodified data in a structured format (ideally as an Excel Table). The Logic tab does all the heavy lifting - transformations, calculations, lookups, using modern Excel functions like FILTER, SORT, and LAMBDA. The Interface tab presents the final, polished output that stakeholders actually see.

This resonates strongly with how we think about data architecture at endjin. But while the original article frames this through the Model-View-Controller pattern, I think there's a more relevant mental model from the data engineering world that fits this use case perfectly.

From MVC to Medallion

If you've worked with modern data platforms - whether that's Databricks, Microsoft Fabric, or Azure Synapse - you'll likely be familiar with the Medallion Architecture. As Carmel described in her recent deep-dive on the topic, it's a data design pattern that consists of three tiers: Bronze (raw), Silver (cleaned and validated), and Gold (projected for specific use cases).

The parallel to the three-tab Excel rule is obvious:

This isn't just a semantic rebrand. The Medallion Architecture brings with it a wealth of best practices from the data engineering community - around data quality, validation, lineage, and governance that can inform how you approach your data workloads, even in Excel.

The benefits of this approach

When you start thinking about your Excel workbook as a miniature data pipeline, several good practices naturally follow.

Data lineage becomes visible. With clear separation, you can trace any value in your Output tab back through the Clean tab to its origin in the Source tab. When a number looks wrong, you know where to look.

Validation can be added systematically. The Clean tab becomes the natural place to add data quality checks - are there unexpected blanks, or duplicates? Do totals reconcile? Are values within expected ranges? This is the equivalent of creating quality gates as data moves from Bronze to Silver.

The Source tab becomes immutable. Just as the Bronze tier in a data lakehouse preserves raw data for historical playback, your Source tab should remain untouched. If new data arrives, append it rather than overwrite. This gives you an audit trail and the ability to reprocess if your logic changes.

Multiple projections from the same source. You might need different views of the same underlying data - one for the finance team, one for operations, one for the board. In the Medallion Architecture, this is exactly what the Gold tier enables. In Excel, you can create multiple Output tabs, all drawing from the same Clean layer.

Proceed with caution

I should be clear that whilst this pattern makes Excel workbooks more robust, more maintainable, and more professional, it doesn't make Excel the right tool for every job. I hold some strong opinions about when and where Excel is appropriate, and when the stakes are high - when significant decisions are being made and errors could impact financial, regulatory compliance, or even public health outcomes, Excel isn't enough. You more than likely need proper software engineering discipline and quality control barriers across technology, process, and people. This pattern doesn't prevent someone accidentally filtering and deleting rows in the Source tab. It doesn't stop formulas breaking silently. It doesn't provide version control, automated testing, or audit logs.

But, for those situations where Excel genuinely is appropriate, or where constraints mean it's your only option, this approach represents a genuine step-change in how you structure your work.

Summary

The three-tab rule for Excel is a pattern worth adopting. It has a direct comparison to the well established Medallion Architecture, connecting your Excel work to the broader principles of modern data engineering: separation of concerns, data quality, lineage, and fit-for-purpose projections.

Of course, this approach doesn't come without limitations. Excel remains a tool designed for individual productivity, not enterprise data management. If your workbook is becoming mission-critical - if it updates regularly, if others depend on it, if errors would be costly, then it's worth asking whether you've outgrown what Excel can safely provide.

The Max operator offers a projection-based overload with this signature:

public static TResult? Max<TSource,TResult>(
    this IEnumerable<TSource> source,
    Func<TSource,TResult> selector);

This will iterate through the source, pass each item to the selector callback, and then return the highest of the values the callback returns.

Notice how although the selector function returns a TResult, the return type of Max itself is TResult?. That nullability is there to handle the case where the source enumerable is empty: in that case there is no maximum value (because there are no values at all) and that TResult? return type means Max can return null to indicate that.

But it goes a bit weird if the selector returns a value type. Suppose you've got this type (which is a reference type, but crucially, two of its properties use value types):

public record WithValues(string Label, int Number, DateTimeOffset Date);

First, let's verify that Max does what I've said with an empty list when the projection retrieves a reference type:

WithValues[] empty = [];
string? maxLabel = empty.Max(x => x.Label);
Console.WriteLine(maxLabel is null);

This prints out True, confirming that Max here does indeed return null to let us know that there was no maximum value. (The notion of a maximum string value raises the awkward fact that Max doesn't let you pass an IComparer<T> here, but let's ignore that for now.)

With that in mind, what type do you suppose maxDate has in this example?

WithValues[] empty = [];
var maxDate = empty.Max(d => d.Date);

If you look at the definition of Max you could correctly conclude that TSource here becomes WithValues and that TResult is DateTimeOffset. (And as we're doing all this type inference in our heads, we might reflect on whether using var here has really saved us any time and effort.) And since Max returns TResult? you might conclude that maxDate must be of type DateTimeOffset? (which is an alias for Nullable<DateTimeOffset>).

But that would be wrong. Here's exactly equivalent code using an explicit type declaration instead of var:

WithValues[] empty = [];
DateTimeOffset maxDate = empty.Max(d => d.Date);

It is now clear that maxDate's type is DateTimeOffset. If we were to try to declare it as a DateTimeOffset?, that would actually compile, but it wouldn't be equivalent to the var example: in the case where we use var, maxDate really does have the non-nullable DateTimeOffset type.

And if we do try to use DateTimeOffset?, it goes wrong. This compiles:

DateTimeOffset? maxDate = empty.Max(d => d.Date);
if (maxDate.HasValue)
{
    Console.WriteLine(maxDate.Value);
}
else
{
    Console.WriteLine("No dates found.");
}

but it only compiles without error because an implicit conversion is available from Max's return type of DateTimeOffset to the variable's type of DateTimeOffset?.

The most important thing to know about this code is that it will actually fail at runtime with an InvalidOperationException complaining that the Sequence contains no elements!

Earlier I linked to a non-generic overload of Max that returns an int? so you might think that this would work:

WithValues[] empty = [];
int? maxNumber = empty.Max(x => x.Number);

but this will also fail with an exception at runtime instead of returning null. In fact it ends up using a different overload that returns an int, and not the one that returns an int?.

So that's weird. The first two examples call the same single overload of Max, and yet it handles an empty list completely differently depending on whether our selector returns the Label or Date. (When it returns Number, we end up using the int-specific overload, but the fact remains that an empty list causes an exception when we select a value-typed property, but the method returns null when selecting a reference-typed property.) What's going on?

Well it turns out that this particular Max method actually has two different code paths: and it effectively uses this test to choose which path to use:

TResult val = default;
if (val == null)
...

If val == null, then it goes down the code path that returns null if the list is empty. If not, it goes down the path that throws an exception if the list is empty.

This is a deliberate design choice. If default(TResult) is something other than null—e.g. default(int) is 0—then there might be no way to tell the difference between an empty list, and a list where default(TResult) really was the maximum value. For example, in the list [-3,-2,-1,0], the maximum value is 0, so how could we distinguish between that case and the empty list case if we were getting back 0 in either case?

So there's a rationale for this behaviour, but it's not obvious that this one method can behave in two quite different ways. The documentation doesn't mention that this particular overload may throw an InvalidOperationException.

We can explore what that test will do with various types:

static void ShowNull<T>()
{
    T? val = default;
    Console.WriteLine(val == null);
}

ShowNull<string>();
ShowNull<string?>();
ShowNull<int>();
ShowNull<DateTimeOffset>();
ShowNull<int?>();
ShowNull<DateTimeOffset?>();

This prints out:

True
True
False
False
True
True

So this tells us that Max will consider TResult to be potentially nullable if it's a reference type like string, or if it's a nullable value type like int? or DateTimeOffset?. But plain value types like int and DateTimeOffset are considered not to be nullable.

That explains why using the x => x.Label lambda makes Max return null when the list is empty, while with d => d.Date or d => d.Number, it throws an exception. The first has a return type of string (a reference type) while the other two have non-nullable value-typed return types (DateTimeOffset and int).

But why does Max even have these two different code paths? It's perfectly possible for a method to return a DateTimeOffset?, so why does Max not do that here? If the argument for the type parameter TResult is DateTimeOffset, and the method declares a return type of TResult?, shouldn't that make the return type DateTimeOffset??

The reason it doesn't work out that way is because nullability handling for reference types was a bit of an afterthought in C#. (See my extensive series on nullable reference types for (a lot) more detail.)

In the beginning (C# 1.0) there were value types, which could not be null, and reference types, which were always capable of being null. There simply wasn't any concept of a value type being nullable, and nor was there any way to constrain a reference type to be non-null. This reflected the underlying reality of the .NET runtime's type system. Then C# 2.0 added support for nullable value types, enabling us to write int?. But this was an entirely different way of representing nullability: an int? is really a Nullable<int>, and Nullable<T> essentially combines a value with a bool indicating whether the value is present. This is fundamentally different from how reference types like string represent null. (This is more of a library feature than a runtime feature. OK, strictly speaking there's some special handling for Nullable<T> when it comes to boxing and unboxing, but otherwise, this is mainly a language feature that doesn't directly reflect how the underlying runtime type system really works.) Although C# lets us write code that works with int? in ways that are (sometimes) similar to how we might work with a reference, the generated code is really quite different, and that causes challenges for generic code. And finally, C# 8.0 introduced nullability annotations for reference types, so that now, we write string? if we mean a reference that might be null whereas string is (in theory) never null, in a way that is conceptually similar to the fact that an int can never be null.

But although we've ended up in a place where there are apparently two dimensions—value vs reference, and nullable vs non-nullable—the history of how we got here means these aren't truly independent. Nullability works very differently for values vs references in practice. And two of the four combinations (nullable values, and non-nullable references) aren't really first class citizens in the .NET type system. (A nullable value in a null state looks different from the null reference value. And a 'non-nullable' reference might in fact be null.)

And this difference tends to poke out from time to time with surprising behaviour like we're seeing with this Max operator. It would be completely reasonable to expect it to deal with the Label and Date properties in exactly the same way. But the history of nullability in .NET means it doesn't work in practice.

So how do we fix this? We can use this slightly ugly hack:

DateTimeOffset? maxDate = empty.Max(d => (DateTimeOffset?)d.Date);

That cast means that the lambda's type is now Func<WithValues, DateTimeOffset?>. (Before it was Func<WithValues, DateTimeOffset>, with a non-nullable return type.) Since default(DateTimeOffset?) == null, Max will select the code path that returns null when the input collection is empty. (It doesn't do that without this cast, because default(DateTimeOffset) is not null. It's a value representing midnight on the 1st January in the year 1, with a zero time zone offset.)

But what about that specialized (non-generic) overload of Max I linked to earlier that returns an int?? Well it turns out that it only comes into play when the selector also returns an int?. You get a different overload when the selector returns a plain int. So it ends up looking similar to the DateTimeOffset case (which used the generic overload). We need to cast to a nullable value:

int? maxNumber = empty.Max(x => (int?)x.Number);

So we can make it work how we want, it's just slightly messy. That's the reality of a 25+ year old language that has made two major changes to the nature of what it means to be null.

When I first arrived at Gatwick airport I had no idea what life was going to look like - I stayed with a friend, found some temporary work, and tried to reacclimatise to life at home. There were many reunions, tears, and more than one panic attack due to complete overwhelm - reverse culture shock is a real thing, who knew?

But, when all was said and done, though I made some vague attempts to see what was out there in the working world, it felt like there was really only one place I wanted to end up - and that was back at endjin.

I've now been back in the UK nearly two years, and back at work for 18 months, and there is definitely a lot to reflect on.

Remote Work and Digital Nomading

I know "digital nomad" can sound a bit buzzword-y, but it is the simplest label for how I have been living since I got back.

Having spent almost 3 years without staying in one place more than a month, the idea of signing a year-long lease felt terrifying in the extreme. And, endjin being a totally remote company (and having been so since 2018 - before it was cool...), I luckily didn't have to.

A lot of Airbnbs offer discounts for stays over a month, and with the current price of electricity and gas, you can often find options for far less than you'd pay at a standard rental - especially if (like me) you are drawn to places in the middle of nowhere, even in the depths of winter. So, that is what I did...

I lived all over the UK - Devon, Yorkshire, Bristol, North Wales, and even spent the winter in Spain (something that convinced me more than ever that January in the UK just isn't for me). And, in all of this, I learnt a lot about what I value in the places that I live. The feeling of being able to walk out the door into nature is something that, for me, is unparalleled. That being said, somewhere with train connections allowing me to attend the 5+ weddings I need to go to (yes, I am in my 30s) is equally important. Two months spent in a depression fog in a village in Yorkshire taught me that access to a gym or at least some way to exercise when it's raining is also a must for my mental health...

Honestly, I do not think returning to work would have been possible for me at that point without this flexibility. My mental state was not great, and trying to go straight from years of constant movement into a rigid routine would have been too big a shock. Being able to choose where I lived, reduce pressure where I could, and make changes gradually meant I could build back up rather than burn out.

Remote-first work gave me the space to re-enter life in the UK on my own terms, and that flexibility is what has made the first year back at work feel in some ways like a continuation of the adventure.

What I Want to Carry Forward

I know that not everyone has the opportunity, means, or even desire to live month-to-month unpacking and re-packing, but I do think that there are some lessons that I've learnt which are applicable whatever your situation:

• Over-planning the next 1/3/5 years can make you more anxious, not less. One of the biggest things I had to accept when travelling was that nothing ever goes fully to plan. That's as true for life in general as it is for catching 4 buses in a day. Spending all my time running through every possible scenario and outcome is never as helpful as it feels in the moment. (As someone with anxiety, I know that's easier said than done. And, to be clear, I'm not saying don't plan at all - I'm told some financial planning is probably a good idea...)
• You don't need to work everything out at once. Trying something and deciding it doesn't work is far better than never changing at all. Most of the time, all you need to plan is the next step.
• Notice what makes you happy. Nothing makes me feel better than being in nature, so building a life around that feels not only logical, but necessary.
• Also notice what drains you. For me, winter has always been hard. Once I accepted that, it became much easier to make decisions that were actually good for me.
• Revisit your priorities every few months. What mattered to you last year might not be what you need now.
• Don't confuse discomfort with failure. Some uncertainty is just part of change, and it does not always mean you've made the wrong decision.

And, if you are considering stepping into the world of "digital nomading", some advice:

• If you are working, make sure that you have enough time to appreciate a place. The first couple of places I stayed, I was only there for a month. By the time I'd moved the first weekend and left the final one, I felt like I had barely found my feet before I was moving again. Plan a good margin between moves - back-to-back travel plus work can be exhausting very quickly.
• Think about the practical - what are some of the things that you do every day / every week that you'd struggle without - a swimming pool? A gym? A library? A pub within walking distance..?
• How much travel do you need to do whilst you are there? Can you find somewhere you love that doesn't mean spending 5 hours on a train multiple times per month?
• Always have a backup internet option. A local SIM/hotspot can save a lot of panic on work days.
• Budget for comfort, not just cost. Sometimes paying a bit more for location, heating, or a proper desk is worth it.
• Remember that it can be lonely. Moving around means you don't always build a base where you are living. Make sure you know what you will do if you are feeling alone - are there friends nearby? Do you have people you can call? Are there local groups you can get involved in?

Final thought

The biggest thing I have learned is that you do not need to blow up your whole life to make things better. You can use the same approach wherever you are: pay attention to what helps, be honest about what drains you, and focus on the next sensible step instead of waiting for a perfect long-term plan. That shift has made life feel less overwhelming, work feel far more sustainable, and I'm excited about building a life that works for me - whatever that might look like!

People often ask questions like "Should I use Cursor or ChatGPT?" or "Is Claude better than Copilot?" but to make decisions about which tools to use (or even ask the right sort of questions), it's important to understand the underlying architecture.

In this post, I want to share a simple mental model that has helped me make sense of the AI-assisted coding landscape.

The four layers

At a high level, we can think of AI-assisted coding as being composed of four distinct layers:

1. Harness: The user interface and experience for interacting with the AI. It includes things like code editors, chat interfaces, CLI tools, and the system prompts that shape the AI's behaviour.
2. Capabilities: The tools, instructions, skills, and context sources that extend what the AI can do — increasingly portable across harnesses.
3. Model: The AI model that processes input and generates tokens — text, code, images, or other outputs. This is where the "intelligence" lives.
4. Provider: The infrastructure and services that host and run the model. This includes cloud platforms, APIs, and the computational resources (GPUs, memory, state) needed to power it.

These layers build on each other: the harness provides the interface, capabilities extend what's possible, the model does the reasoning, and the provider supplies the compute.

Harness

The harness is what you actually interact with day-to-day, and there's a surprisingly wide range of options. Broadly, they fall into a few categories:

• Chat interfaces: Web-based or desktop tools like claude.ai or ChatGPT, where you paste code in and get responses back. Great for quick questions and exploration, but limited when it comes to working with full projects.
• IDE extensions: Tools like GitHub Copilot Chat or Roo Code that plug into your existing editor (VS Code, Visual Studio, JetBrains, etc.). These meet you where you already work, with direct access to your codebase.
• Purpose-built IDEs: Editors like Cursor and Google Antigravity that have been built from the ground up with AI at their core (usually forks of VS Code). They offer deep integration between the editor experience and the AI capabilities.
• App builders: Tools like Lovable that focus on generating entire applications from natural language descriptions, targeting less technical users or rapid prototyping.
• CLI tools: Command-line agents like Claude Code, OpenCode, and Copilot CLI that let you work with AI directly from your terminal. These tend to appeal to developers who prefer keyboard-driven workflows.

Even within these categories, there has been a push recently towards different user interactions. For example, integrating voice control or continuing conversations across devices.

But the harness isn't just about where you interact with the AI, it's about how the harness shapes the AI's behaviour. Modern harnesses go well beyond simple chat, adding capabilities such as:

• Agentic workflows: The ability for the AI to plan, execute multi-step tasks, spawn sub-agents, run commands, and iterate on its own output. This might happen locally in the harness, or it might hand off to cloud-based agents running on the provider layer.
• System prompts: The invisible instructions that shape how the AI behaves. This is a bigger deal than most people realise. The same model can perform dramatically differently depending on the harness it's running in, because each harness ships its own system prompt.
• Memory and context: Persistent memory across sessions, project-level instructions, and the ability to pull in relevant files and documentation automatically.

These harness-level capabilities can make a huge difference to your productivity, often more so than the choice of model itself. A great model in a limited harness won't perform as well as a good model in a harness that gives it the right tools and context.

Capabilities

The capabilities layer is what you layer on top of the harness to extend what the AI can do. This has arguably been one of the biggest developments in the last year. It includes:

• Tools and MCP: The Model Context Protocol (MCP) has emerged as a standard way to give AI access to external tools — running tests, querying databases, calling APIs, searching the web, interacting with design tools, and more. These tools are increasingly portable: the same MCP server can work across Claude Code, GitHub Copilot, Cursor, and other harnesses.
• Instructions and skills: Project-level instruction files (like .instructions.md or .cursorrules) that tell the AI about your codebase conventions, preferred patterns, and how to approach tasks. Custom skills and agent definitions let you package domain-specific knowledge that the AI can draw on.
• Context sources: Documentation, codebase indexing, knowledge bases, and other reference material that help the AI understand your specific domain and codebase.

What makes capabilities a distinct layer (rather than just a feature of the harness) is their portability. You can take the same MCP servers, the same instruction files, and in many cases the same context sources, and use them across different harnesses. Your investment in configuring capabilities isn't locked to a single tool.

Model

The model is where the "thinking" happens. When you send a prompt, it's the model that interprets your intent, reasons about the problem, and generates tokens in response — whether that's code, prose, or increasingly other modalities like images and audio. Models differ across several key dimensions:

• Reasoning ability: How well the model can break down complex problems, plan multi-step solutions, and handle nuanced logic. The emergence of dedicated reasoning modes (like "extended thinking") has been a significant step forward here, though higher reasoning levels consume substantially more tokens.
• Code generation quality: The accuracy, correctness, and idiomatic quality of the code it produces across different languages and frameworks.
• Tool use: How reliably the model can decide when and how to call external tools provided by the harness and capabilities layer, and how well it can structure its output to work with those tools.
• Context window: How much text the model can "see" at once. Larger context windows mean the model can work with bigger codebases without losing track of important details.
• Speed: How quickly the model generates responses. For interactive coding, latency matters, so a slower but more capable model isn't always the best choice for every task.

Today's landscape includes several categories of model:

Frontier models - like Claude, GPT, and Gemini - are the most capable, hosted in the cloud, and accessed via API. They're constantly being updated and represent the cutting edge.

Local models can run on your own hardware. Tools like Ollama or LM Studio make it straightforward to run open-weight models (e.g. Qwen, Llama, DeepSeek). They're typically less capable than frontier models, but they offer advantages in terms of privacy, cost (no per-token charges), and the ability to work offline. It's worth noting that "open-weight" doesn't always mean fully open — you can download and run the model, but you typically don't know how it was trained or on what data.

Specialist models are smaller, narrowly focused models tuned for specific tasks: speech-to-text (e.g. Whisper), text-to-speech, classification, summarisation, OCR, and more. While frontier models are incredibly powerful, they're also expensive; for high-volume business tasks, smaller and more cost-efficient models often make more sense.

And you don't have to pick just one. Many harnesses let you switch models on the fly, so you can use a fast, lightweight model for simple tasks and a more powerful frontier model when you need heavy reasoning.

Provider

The provider layer is often invisible to individual developers, but it's where many of the most important enterprise concerns live. When an organisation is evaluating AI coding tools, the questions at this layer tend to dominate the conversation:

• Data residency: Where are your prompts and context being sent, and where are they processed? For regulated industries, data staying within a specific geographic region can be a hard requirement.
• Security and compliance: Does the provider meet the organisation's security standards? Are prompts and code snippets logged or used for training? What certifications does the provider hold (SOC 2, ISO 27001, etc.)?
• Copyright and IP: There are risks associated with generating code using models trained on public data, or agents retrieving proprietary information. Some providers offer guarantees around IP ownership and indemnity (such as Microsoft's Copilot Copyright Commitment), which can be crucial for commercial use.
• Rate limits and availability: How many requests can you make before hitting throttling? Is there an SLA for uptime? For a team of developers relying on AI throughout the day, rate limits can become a real bottleneck.
• Cost management: Pricing varies significantly, from flat-rate subscriptions to per-token usage billing. At scale, understanding and controlling costs becomes critical. This is compounded by the cost of reasoning: enabling higher reasoning levels on capable models can dramatically increase token consumption.

This is where options like Microsoft Foundry, and Amazon Bedrock come in. They let enterprises access frontier models through their existing cloud provider (though not always within the same infrastructure), with the governance, networking, and compliance controls they already have in place. You get your own dedicated capacity, and billing flows through your existing agreements. Model marketplaces like Hugging Face also play a role, providing a catalogue of models (both open-weight and commercial) that can be deployed on your own infrastructure.

For most individual developers, the provider layer is something you don't think about much — it just works. But for teams and organisations adopting AI coding tools at scale, it's often the layer that determines which tools are actually allowed to be used.

Bundled vs. mix-and-match

In practice, you'll see these layers packaged together in different ways. Some products bundle all layers tightly, while others give you the freedom to pick and choose.

For example, a Claude Pro subscription bundles everything: the claude.ai chat interface, Claude desktop app, and Claude Code CLI (harness), Anthropic's Claude models (model), and Anthropic's own infrastructure (provider). It's a clean, simple experience — but you're largely locked into Anthropic's choices at every layer.

GitHub Copilot takes a more flexible approach. You get VS Code / Visual Studio IDE extensions (or Copilot CLI) as your harness, but you can choose from a wide selection of models: Claude, GPT, Gemini, and others. The models are hosted on different providers behind the scenes, but this is abstracted away. You can also bring your own capabilities via MCP servers and instruction files. You can even use alternative harnesses like the Claude SDK from within the Copilot ecosystem, or connect local models.

Then there are tools like OpenCode or Roo Code, which are open-source harnesses that let you bring your own model and your own provider. You could run a local model on your own hardware, connect to an API key with OpenAI, or point it at an Azure OpenAI deployment your team manages. This gives maximum flexibility, but you're responsible for wiring it all up.

Blurring boundaries

It's worth noting that the boundaries between these layers are starting to blur. Agentic capabilities that used to live purely in the harness are increasingly being pushed into the model and provider layers. Code can run on your local device, in the cloud, or as a fleet of sub-agents. And you can orchestrate all of it from your IDE, a terminal, or even a mobile device. The mental model is still useful for making decisions, but the sharp lines between layers are softening as the ecosystem matures.

Conclusion

This post isn't intended to recommend a specific tool or combination - what's right for you will depend on your constraints, your team's needs, and the kind of work you're doing. But by understanding that there are four distinct decision points - harness, capabilities, model, and provider - and the trade-offs at each layer, you can make informed choices rather than getting lost in the noise. When someone asks "Should I use Cursor or Claude?", you'll know that's not quite the right question, and you'll know what questions to ask instead.

In the previous posts in this series, we introduced the Corvus.Testing.AzureFunctions.ReqnRoll project and showed how you can use the bindings and classes it provides to start functions apps as part of your scenarios and features. We're going to finish with some pointers on how to ensure these functions can run as part of your build pipelines.

Depending on how your build system works, it's relatively easy to ensure that tests using these methods are able to run as part of the build pipeline.

If, like us, you're using Azure DevOps with hosted agents, you'll need to add a step to your pipeline to install the Azure Functions Core Tools. For YAML build definitions, it looks like this:

- task: Npm@1
  displayName: 'Install Azure Functions V4 Core Tools'
  inputs:
    command: custom
    verbose: false
    customCommand: 'install -g azure-functions-core-tools@4 --unsafe-perm true'

Once that step has run, the test will be able to execute as it does locally. The Corvus.Testing.AzureFunctions.ReqnRoll library targets .NET 8, which is cross-platform, so this should work on both Windows and Linux build agents.

GitHub Actions is similarly straightforward. Simply add a step to install the Azure Functions Core Tools towards the start of your pipeline.

This approach should also work for other CI servers and their hosted build agents. If you're using a private agent you have the option of installing the tools globally, meaning your build scripts can just assume they are present - this very much depends on how you prefer to manage build agents.

Summary

For Reqnroll users, the techniques I've shown in these posts will help ensure your integration tests are more complete by ensuring that functions under test are hosted in a way that closely matches the Azure environment they will ultimately run in.

I've tried to keep the posts simple by only covering testing HTTP triggered functions, but these techniques can equally be used to test functions that use other trigger types too. In our projects we've used it to test functions with blob, queue and Event Hubs endpoints, as well as functions using Durable extensions. This will generally require other infrastructure for your integration testing - for example, Azurite and/or Testcontainers to make storage available to trigger blob and queue endpoints - but the principles remain the same.

As mentioned above, the Corvus.Testing projects (Corvus.Testing.ReqnRoll and Corvus.Testing.AzureFunctions.ReqnRoll) are open source and contributions are accepted. If you encounter problems with them, please feel free to raise an Issue - and if you're able, submit a pull request. And if you have any questions or feedback, just ask!

In the previous posts in this series, we introduced the Corvus.Testing.AzureFunctions.ReqnRoll project and showed how you can use the bindings and classes it provides to start functions apps as part of your scenarios and features. Here, we look at how you can vary the behaviour of those functions apps by providing or overriding configuration values.

When the FunctionsController is used to start a new function, it will check the ScenarioContext (if available) and FeatureContext for an instance of the FunctionConfiguration class. Any configuration provided here will be made available to the functions app when it starts.

If you're using step bindings to start your function, you can do this by writing an additional step binding to provide configuration from wherever you need to retrieve it. Note that this step must come before the one that starts the function. You can see an example in ScenariosUsingStepBindings.feature:

If you're using a BeforeScenario or BeforeFeature hook, you can add the configuration at the same time - as shown in ScenariosUsingPerScenarioHookWithAdditionalConfiguration.feature (and the corresponding hook method in DemoFunctionPerScenario, StartFunctionWithAdditionalConfigurationAsync), as well as the per-feature equivalents.

If you need to use different configuration at different times, you can create separate hook methods for setting up configuration - just ensure you use the Order parameter on the hook attributes to ensure the configuration is set prior to the functions being started.

In the first two posts in this series, we introduced the Corvus.Testing.AzureFunctions.ReqnRoll project and showed how you can use the Reqnroll step bindings it provides to start functions apps as part of your scenarios. This approach has the drawback of making your scenarios harder to read for non-technical users, so in this post, we're going to take a look at using scenario and feature hooks to address that problem.

Using per-scenario hooks to start and stop functions - as shown in ScenariosUsingPerScenarioHook.feature

Reqnroll hooks allow us to add code that's executed at specific points during a test run. With this method, we make use of the BeforeScenario and AfterScenario hooks, and use the Corvus.Testing.AzureFunctions.FunctionsController class directly to start and stop our functions. This can be seen in the DemoFunctionPerScenario class:

The parameters that the StartFunctionsInstance method takes are the same as those shown in the step binding above, allowing you to specify project, port and runtime. You'll see that the created FunctionsController instance is stored in the ScenarioContext; this allows us to pull it out in the AfterScenario method (which you should add yourself, as shown in the test code) to tear down the functions.

Advantages to this method

Using this method conceals the technical detail of what the setup step involves, reducing it to a single tag for the function. This makes your scenarios much more readable. If you're writing lots of tests for a specific functions app, it also reduces the duplication needed when every scenario has to contain the setup step.

In addition, test output (and the associated functions output) can be viewed in exactly the same way as above.

Disadvantages to this method

The main disadvantage to this approach is one that's associated with pretty much all integration testing: speed. Whilst setting up and tearing down all dependencies for each test is the gold standard, spinning up functions takes time and this can mean your test suite takes a long time to execute. In some scenarios this may be unavoidable. However, others may lend themselves to using the third method to strike a balance between speeding up execution and isolating tests.

Using per-feature hooks to start and stop functions - as shown in ScenariosUsingPerFeatureHook.feature

Visually, this approach looks extremely similar to the previous method. The scenario definitions are not affected at all and the only differences in the underlying code (other than using BeforeFeature and AfterFeature attributes) being that the FeatureContext is used in place of the ScenarioContext to store and retrieve the FunctionsController. The other difference is that the hook methods themselves need to be static to be used with per-feature hooks - this is a Reqnroll requirement.

Advantages to this method

If you can group related scenarios and be sure they won't conflict with one another, this is a relatively easy way of speeding up test execution.

Disadvantages to this method

As implied above, this approach does have the potential to cause unexpected results if your tests conflict with one another in any way. The other disadvantage is that because the function output is gathered and written to console when the function is terminated, it can no longer be seen in the test output. If you don't mind duplication, you can get round this by adding an additional AfterScenario hook to write the function output to the console after every scenario.

In the first post in this series, we introduced the Corvus.Testing.AzureFunctions.ReqnRoll library. In this post, we're going to take a look at the simplest way of using it to start functions apps for testing purposes, which is to use the provided step bindings.

This is demonstrated in ScenariosUsingStepBindings.feature in the Corvus.Testing.AzureFunctions.ReqnRoll.Demo.Specs project.

The Corvus.Testing.AzureFunctions.ReqnRoll project contains a Given step definition for the following pattern:

[Given("I start a functions instance for the local project '(.*)' on port (.*) with runtime '(.*)'")]

If you include steps that match this pattern in your scenario, they will cause the functions defined in the specified project to be run, with HTTP functions listening on the specified port. If your function doesn't actually have any HTTP endpoints you can supply a dummy value for the port. Runtime will most likely be net8.0 (for Functions v4), though net9.0 and net10.0 are also options depending on your target framework.

The project to run is currently resolved by traversing up the folder tree until it finds a folder that, when combined with the function name, runtime and build folder, provides a valid path.

As well as bindings for these steps, there's an additional AfterScenario hook that goes with them to tear down the functions instances they start. You can start multiple functions in a single scenario using these bindings if necessary.

Viewing function output

Once the test run is complete, output from the functions app can be seen in the Test Detail Summary. In Visual Studio, this is visible in the Test Explorer by selecting the scenario that's been executed and clicking the scenario that's been selected:

Clicking the link "Open additional output for this result" will show Reqnroll's standard output capture:

As you can see from the screenshot above, this starts with the output from the BeforeScenario binding showing the solution and runtime location. If starting the function failed for some reason, you'd most likely see the reason here.

This is followed by Reqnroll's standard per-step output. Finally the output from the AfterScenario binding is shown, which is where the StdOut and StdErr for each function is added.

Note that the log shown in this window is frequently a truncated version of the whole. If this is the case, you'll see a message explaining how to access the full log by copying and pasting into another tool.

Advantages to this method

Using step bindings in this way makes it crystal clear to the developer what's going on as part of their spec. You can easily see which functions are being run and on what ports.

Disadvantages to this method

Whilst it's nice for developers to see exactly what technical setup is taking place, this does go against the goals of Behaviour Driven Development. Specifically, we should be striving to make the feature readable in the end user's language. When testing an API using a BDD spec, you can make a case that the end user whose language we should be using is a technical one - the consumers of APIs are most likely to be developers - but even so, this is an overly technical step to include in your scenarios.

If you use Azure Functions on a regular basis, you'll likely have grappled with the challenge of testing them. The testing story for functions is not hugely well defined. If you're building your functions well, then there won't be a lot of code in them - they will be thin facades calling into code that does the bulk of the work, in which case you will likely have used a standard unit testing approach on that code. Nevertheless, that likely leaves some functionality untested - for example, ensuring your models are correctly bound to input, and ensuring that correct status codes, headers and so on are returned from your requests.

As such, it becomes necessary to step up a level and look at how to test the functions as a whole. There are two options for this:

• You can test in-process, using the approach defined in Microsoft's docs (note that this doesn't seem to have been updated to take account of functions that use instance methods, but that's unlikely to affect the approach). This is good, but doesn't ensure that your function is configured correctly, and if you're using automatic model binding, it doesn't test that this is working as you expect.
• Alternatively, you can test out of process, either against a deployed instance of the function or against one that's running locally. Testing against a deployed instance is a great idea, but this is normally reserved for another level of testing, meant to ensure that things are working as expected in a deployed environment. It doesn't address the needs of the developer as the feedback loop from making a change to deploying a function to Azure is likely just too long. This leaves us with the challenge of testing against a function running locally.

So, how do we go about this?

Before I continue I should note that while I'm specifically addressing how to do this with Reqnroll, a very similar approach can be taken with other frameworks. Reqnroll is the community-driven successor to SpecFlow, created by the original SpecFlow creator after SpecFlow reached end-of-life. If you're migrating from SpecFlow, the Reqnroll migration guide is a great place to start.

Goals

As always, it's worth starting with what we want to achieve:

1. We want a way of automatically starting a function, and then shutting it down once the test is completed.
2. We want this to work in as close a way as possible to a deployed function
3. Ideally, we want to be able to capture the output from the function while it's running.
4. It's useful to be able to easily affect the configuration of the function under test.
5. We want an approach that can work as part of a CI pipeline.

So, let's have a look at how we achieve these goals.

Running the function locally

When you hit F5 to run a function in Visual Studio, it uses a copy of the Azure Functions Core Tools that's managed by Visual Studio. Normally they get automatically installed into C:\Users\username\AppData\Local\AzureFunctionsTools\Releases and Visual Studio selects the correct version to use based on your project's runtime.

However, this is an internal detail of how Visual Studio implements the Functions SDK, so it's not really something we can rely on. Fortunately you can install and use Azure Functions Core Tools directly.

We recommend using Azure Functions v4 with the isolated worker model and .NET 8 or later. The isolated worker model is the recommended approach for new Azure Functions projects, and in-process support is scheduled to end in November 2026.

To get the tools installed, you have a few choices. If you're on Windows, you can use winget:

winget install Microsoft.Azure.FunctionsCoreTools

or Chocolatey

choco install azure-functions-core-tools

Otherwise, you'll need npm:

npm i -g azure-functions-core-tools@4 --unsafe-perm true

This will install the tools locally - you can verify they are there using the new func command from the command prompt. If you do this, you'll see all the things you can do with it - scaffolding new functions apps and functions, and running functions locally. The latter is what we're concerned with - you'll see that you can start a new function using the command func start, providing port number and other details as part of the command. This is what we're going to use when setting up our test.

Introducing Corvus.Testing

The code to start, stop and manage functions as part of a Reqnroll test is part of the endjin-sponsored Corvus.Testing libraries. The original Corvus.Testing repository has been split into separate, focused repos:

• Corvus.Testing.ReqnRoll — general Reqnroll testing utilities
• Corvus.Testing.AzureFunctions.ReqnRoll — Azure Functions-specific testing classes and bindings

The classes that we're interested in are part of Corvus.Testing.AzureFunctions.ReqnRoll and are:

FunctionsController.cs - this contains methods to start a new functions instance, and to tear down all functions it manages. It's intended to live for the lifetime of the test as it captures the output and error streams from the function and write them all to the Console when the functions are terminated. When running in Reqnroll, this results in that information being written to the test's output.

FunctionConfiguration.cs - this is part of the mechanism by which the test project can provide settings to the function under test.

FunctionsBindings.cs - this provides a couple of standard step bindings that can be used as part of a scenario to start a function.

This code is all open source, and contributions are accepted. It's available under the Apache 2.0 open source license meaning you're free to use and modify the code as you see fit. The license does impose some conditions around retaining copyright attributions and so on - you can read the full details here.

This code ticks the boxes for the first four of the five goals I set out above, providing mechanisms to keep functions running for the duration of test execution, as well as a way to supply additional configuration. The next few sections explain the different ways of using this.

I'll be doing this with reference to the demo projects that are part of the Corvus.Testing.AzureFunctions.ReqnRoll codebase. Before continuing, I recommend downloading the project so you can examine the code. There are two demo functions projects — Corvus.Testing.AzureFunctions.Demo.InProcess for the in-process model and Corvus.Testing.AzureFunctions.Demo.Isolated for the isolated worker model — that contain a slightly modified version of code that's generated when you create a new HTTP-triggered function in Visual Studio. They accept GET and POST requests, looking for a parameter called name in either the querystring or request body, and returning a configurable string containing that parameter.

It also contains a Reqnroll test project, Corvus.Testing.AzureFunctions.ReqnRoll.Demo.Specs containing feature files which relate to the following next few posts in this series.

The quality challenge in the age of AI

Over the last year or so we have seen the world move at 100mph, with AI integration into every product (whether it's a good fit or not!) and organisations eager to deploy LLMs and AI services into production in their own systems and workflows. With frontier models readily available through cloud APIs, and deep enterprise integration into big data platforms through wrapper/portal services like Microsoft Foundry, it's very easy to get started and get excited about what's possible.

This is exacerbated by the fact that AI is now pretty much ubiquitous to all consumers of digital products. If business stakeholders are used to being able to get results instantly from native LLM app interfaces on their devices, then expectations are high right from the offset.

But there's a big gap between a PoC and a working, reliable enterprise solution. On the face of it, adding an LLM service to your application is just another API integration, but their behaviour brings with it a new set of engineering concerns that can cause problems if not understood fully.

This post explains what those things are, and why they should be treated the same as any other engineering quality concern, so that you can build AI-integrated solutions with confidence.

A timeline of engineering quality

Before we dig into the specifics of LLM models and AI services, it's useful to look back at other technology and software architecture patterns, and understand how we thought about them in terms of ensuring quality. The easiest way for me to do that is to look back at my own career. Clearly this won't be a fully comprehensive guide, but there's enough experience in there to highlight commonalities across technology stacks and ecosystems.

Establishing the foundations

When I began working as a software developer in 2001, after a brief dabble with Borland Delphi and ASP pages, I quickly found myself immersed in the world of .NET. This was .NET 1.1 territory and ASP.NET Web Forms (around which there were a lot of strong opinions!). As a framework that was designed to magically generate HTML web pages using a set of server-side components, it was inherently difficult to pull apart business logic from the user interface layer, which made it hard to test.

But what we were building was complicated (a government funded, UK-wide, multi-model journey planner, pre-Google Maps) and we needed to prove that it was working correctly. The "auto-magic" that the framework provided allowed for rapid development of things that were simple, but started to get in the way and make things harder as the logic and interactions became more complex. I learned about unit testing as these tools started to become available for .NET (NUnit, MbUnit and then MSTest), and how to refactor code to be able to validate the things we needed to. Having confidence that things were working as they should shifted the dial from slow and brittle feedback loops to rapid, reliable validation cycles.

By the time the ALT.NET movement gained traction towards the end of the 2000's, I was a full-blown TDD aficionado. Shifting the focus to test-first made for better system design, and along with it came more advanced techniques and tooling like mocking, inversion-of-control containers and continuous integration processes to automate quality gates.

Around this time, I was also lucky enough to attend one of JP Boodhoo's .NET engineering bootcamps in Vancouver, which embedded the value of executable specifications with Behaviour Driven Development (BDD), highlighting the importance of natural language and encouraging closer collaboration between business and technical teams.

These core principles have been the foundation underpinning all software development I've been involved in since, enabling high-quality delivery of well documented code in iterative development cycles. They've been applied across web and native application stacks and across a variety of architecture patterns. But whilst there's been specific implementations and frameworks that became the flavour of the month according to the language or toolset of the moment, it's the concepts and approach that have always been the thing.

Contract-first, observability and resilience in the cloud

Fast forward to the 2010s and the world had moved on to API-first, REST-based architectures. At this point I was leading a small team responsible for building the payment processing engine for a large Middle-Eastern airline. With these APIs any mistakes could literally cost money, so as well as ensuring that we had comprehensive test coverage, we also focused on instrumentation and observability to help us diagnose things in our production environment. This was how I learned (the hard way!) that not all currencies have 2 decimal places! And more generally that if you're building an API, you also need a way to execute it. This was pre-Postman, and pre-Swagger, so the only thing to do was build our own version of an API client - a test harness that could be used to execute and validate the various endpoints.

In parallel to this, we were moving everything into the cloud, and started to encountered a new set of quality challenges. Distributed systems brought transient failures, eventual consistency, and the need for sophisticated retry logic.

This experience taught me that integration testing isn't just about verifying that your code works, it's about understanding the contract between systems, the innate behavioural patterns and capabilities of those systems, and building evaluation harnesses that can systematically validate behaviour across different scenarios.

Data, machine learning, and the challenge of uncertainty

By the time we get to 2015, the cloud had also allowed us to capture a lot more data. And so the landscape shifted again with an increasing focus on data engineering - machine learning, data science, cloud data platforms and advanced analytics. The data space was, and in many ways still is, less mature when it comes to engineering practices. But whilst the challenges were different, I found myself still applying the same "there's always a way to test something" mentality.

With machine learning and data science, the tendency to draw conclusions from patterns in data that are really just random noise could be balanced with structured experimentation, upfront definition of success metrics and rigorous validation.

Cloud data pipelines that were asynchronous and long-running meant you couldn't just run a quick unit test and get instant feedback. So we needed new approaches - schema validation to catch structural changes and data quality issues early, synthetic data generation to test edge cases that might not appear in production for months and data snapshot testing to validate consistency of pipeline outputs over time. We developed approaches for testing data quality at scale, validating not just that pipelines completed successfully, but that the data they produced was fit for purpose.

The key insight from this is that uncertainty doesn't mean untestable, it just means we need different validation strategies. This sometimes meant shifting from testing specific values to testing behaviours and patterns. My talk at SQL Bits in 2024 - "Do those numbers look right?" summarises how we were thinking about engineering quality in our data solutions, and deep dives into practical approaches for testing Power BI reports, data pipelines and Spark and Python interactive notebooks.

So are LLMs really any different?

Yes and no. On the one hand, they're just another integration - either via an API, or through local model deployment. But on the other hand, they exhibit characteristics that require us to think differently about validation.

They're non-deterministic: Unlike traditional APIs where the same input always produces the same output, LLMs can generate different responses each time due to their probabilistic nature (even when you set the temperature to 0). This makes traditional unit testing approaches which rely on exact output matching ineffective.

They can hallucinate: LLMs can confidently generate plausible-sounding but false information. Unlike a database query that either returns valid data or throws an error, an LLM might return a well-constructed response that is actually wrong - syntactically correct, but semantically and factually incorrect. This requires us to validate not just the structure of responses, but their factual accuracy and relevance.

They produce qualitative, unstructured responses: Traditional software returns structured data (JSON objects, numerical values, boolean flags etc.). LLMs return natural language, which is inherently ambiguous and context-dependent. Despite advancements in the area of structured outputs, this still isn't 100% reliable. And how do you write an assertion that validates "the response should be friendly and helpful"?

However, none of these challenges are entirely new. Non-determinism shows up in async operations, race conditions, rate limiting, or time-dependent behaviour. User experience validation can be qualitative in nature. And we've had to deal with integration points that might return unexpected results.

On that basis we can, and should, apply the same core engineering principles we've always used, albeit adapted to the unique characteristics of LLMs.

1. Break open the black box

Just as ASP.NET Web Forms made it hard to test by tightly coupling UI and logic, LLM integrations can become black boxes if we treat them as magic, closed systems. The solution is the same - refactor to separate concerns. This means treating your LLM interaction as a discrete component with clear inputs and outputs. Which are the configurable bits that you have control over (e.g. the model version, the prompt, the temperature etc.), and which bits are "inside the box" (e.g. the inner workings of the model, system prompts etc.)?

For example, don't embed prompts directly in your application code. Instead, create a prompt management layer that allows you to version, test, and iterate on prompts independently of your application logic.

At endjin, when we build LLM-powered solutions, we structure them so that the prompt construction, model invocation, and response parsing are separate, testable components. This helps to unlock the ability to swap models, adjust parameters, or refine prompts without touching core business logic.

2. Embrace natural language as a feature

One of the biggest insights from Behaviour Driven Development was that natural language specifications bridge the gap between business stakeholders and technical teams. LLMs flip this on its head - natural language isn't just the specification, it's also the programming interface.

This could be seen of as an advantage. You can write evaluation criteria in plain English: "The response should identify the customer's primary concern", "The summary should be under 100 words", "The sentiment should be appropriate to the context". Then you can use LLMs themselves to evaluate these criteria. Don't forget that the LLM can also be instructed to return additional numerical or categorical information that can augment the natural language response (for example a confidence level between 0 and 1), which can enable more traditional testing to still be performed.

Taking it a step further, part of the specification could be a feedback loop to improve the specification (akin to getting someone else to review your work) before you execute the steps. This might mean LLMs all the way down, but in a good way.

The key is to be systematic about it. There's possible weaknesses around this approach when you consider the vagaries of language, but creating a feedback loop to explore the context with an LLM can be very powerful. Define your success criteria upfront, create evaluation prompts that assess those criteria, and validate them against labelled examples before you rely on them in production.

3. Test-first, even for prompts

The discipline of Test-Driven Development teaches us to think about desired outcomes before implementation. This is even more important with LLMs, where it's easy to iterate endlessly on prompts without a clear definition of success.

Start by defining your test cases - specific inputs with expected behaviours. Not exact outputs (remember, we're dealing with non-deterministic responses) but behavioural expectations. For a customer service chatbot, you might want to identify what a complaint is about and make sure the right resolution is offered.

For example:

Given the customer comment is "Despite paying extra for speedy postage, the promised delivery date was missed by 3 days!"
When the chatbot generates a customer service response
Then the response should identify the core issue as 'delivery delay'
And the tone should be 'apologetic'
And the resolution offered should be 'refund of premium postage'

Then iterate on your prompt design until your system reliably meets these criteria. Track your success rate over time. If you're getting 85% success on your test suite, that's a quantifiable baseline you can work to improve. This is infinitely better than the "it seems to work pretty well" approach.

4. Build or use evaluation harnesses

Just as we built and used custom API clients to test our APIs, we need evaluation harnesses for LLM integrations. The good news here is that lots of AI developer services have eval frameworks built in. These aren't just test scripts, they're tools that allow you to systematically evaluate performance across multiple dimensions.

Your evaluation harness should:

• Run your prompts against a diverse set of test inputs
• Capture and version the outputs for comparison
• Apply multiple evaluation metrics (accuracy, relevance, tone, safety)
• Track performance over time as you refine prompts, change models or a model version is incremented
• Handle transient failures and rate limits gracefully

There's comparisons here with machine learning models that are trained / used in Data Science experiments. Whilst you can't do input / output testing, you can test acceptable tolerance ranges, error rates etc.

5. You need labelled datasets

Just as you can't train a machine learning model without labelled data, you can't validate an LLM integration without example inputs and expected behaviours. Even a small amout of well-chosen examples can provide meaningful validation. But also focus on edge cases and failure modes. What happens when the input is ambiguous? When it's in a different language? When it contains unusual formatting or special characters?

As your system matures, invest in building a larger, more diverse evaluation dataset. Involve domain experts that can set the acceptance criteria of the system. This becomes your regression test suite, allowing you to confidently refine prompts or change models while ensuring you haven't broken existing functionality. Given the volume of tests that will be required, don't underestimate the the level of effort required to curate them. This contributes to the TCO of the solution - production grade LLM solutions are expensive endeavours.

6. Embrace transient failures and build resilience

LLM APIs, like any external service, can experience transient failures, rate limits, performance degradation or varying response times. Your integration needs to handle these gracefully.

Implement retry logic with exponential backoff. Cache responses where appropriate. Monitor latency and error rates. Build fallback strategies for when the LLM service is unavailable. These are the same patterns we use for any cloud API integration , just applied to a different integration point.

7. Test behaviours, not values

Finally, apply the same patterns for validating data quality - focus on behaviours and patterns rather than specific values. Instead of asserting that a generated email contains the exact phrase "Thank you for your inquiry", check that it:

• Addresses the customer's stated concern
• Maintains an appropriate tone
• Includes relevant next steps
• Doesn't contain factual inaccuracies

This is more robust to the natural variation in LLM outputs, while still catching the failure modes that matter. It's often beneficial to use a separate LLM model or service to do this validation so that you avoid the innate bias used in generating the original output being used to validate it (i.e. don't ask an LLM to mark its own work). This technique is referred to as LLM-as-a-judge.

8. Instrument and observe

Across all of these approaches, there's a common thread - you need visibility into what's happening. As more work is handed to the LLM, this accentuates the need for human oversight, supervision, quality checks, curation of more examples, responding proactively to new situations (e.g. a new type of enquiry driven by external factors). Instrument your LLM integrations thoroughly. Log inputs, outputs, and evaluation metrics. Track latency, error rates, and cost. Monitor for drift in model behaviour over time.

At endjin, we treat observability as a first-class concern for LLM integrations, just as we do for any other production system.

Summary: From black box to engineering discipline

The journey from prompt engineering to AI programming is really a journey from treating LLMs as magic to treating them as engineered components. Yes, they have unique characteristics (non-determinism, hallucinations, qualitative outputs), but these aren't deal-breakers, they're just new constraints that require adapted approaches.

To succeed in deploying reliable, enterprise-grade AI solutions, treat LLM integrations with the same engineering standards that you apply to any other system component. That means systematic evaluation, defensive engineering, instrumentation, and continuous improvement.

At endjin, we've seen this transformation happen with our clients. The shift from "let's see if this prompt works" to "here's our evaluation framework and current performance metrics" represents a fundamental change in how teams think about AI integration. And it's that shift that enables moving from demos to production-ready solutions with confidence.

In subsequent posts, I'll dive deeper into the practical tools and frameworks that make this systematic evaluation possible.

Sample code to go with this blog post can be found in GitHub

The Problem: When Sequential API Calls Take Days

It's common in data platforms to need to acquire data from HTTP based APIs. If written well, these APIs will be reliable, fast and will allow you to control filtering, windowing, and pagination. But even if all of this is true, ingesting large amounts of data via an API can be challenging.

We recently faced one of these challenges when building out a new workload on an existing Azure Synapse-based modern data platform: the initial load of data from a source system required us to ingest about 2,000,000 records and the only way of doing this was via an HTTP API. A full synchronization requires tens of thousands of individual API requests, each fetching a page of 100 records of data.

We started by estimating how long this would take with a simple sequential approach. The API averages about 500ms per request. For 20,000 API calls:

20,000 requests × 0.5 seconds = 10,000 seconds = 2.8 hours

We also need to consider the volumes of data being processed. For us, the payload size is generally from 200KB to 1.5MB, with an average of 1MB. So 20,0000 requests yields around 20 GB of data.

That seems manageable, right? But this calculation ignores several real-world factors:

• Network variability: Some requests take 2-3 seconds, especially during peak hours
• Failures and retries: Network issues and transient API errors require retry logic
• Data processing: Each response needs parsing, transformation, and storage

In reality, a test retrieval of a subset of records suggested we were looking at something in the region of 48 hours to do a full ingestion, accounting for failures and retries.

That's too long. Our requirements were clear:

• Scalable: Process massive volumes efficiently through parallelism
• Fault-tolerant: Handle API failures gracefully without losing progress
• Fast: Complete full ingestion in under 12 hours

Why Not Use Synapse Pipelines?

The customer in this scenario already had a well established Azure Synapse platform in place. As a result the first port of call for data ingestion is normally a Synapse pipeline. After all, Synapse is designed for data orchestration, has many built in connectors and is already fully integrated with our data lake. However, we quickly discovered several dealbreakers for this specific use case:

Cost Inefficiency

Synapse pipelines charge per activity execution (approximately £0.00085 per activity run) and per integration runtime hour. Let's break down the cost for our 40,000 API calls:

20,000 activities × £0.00085 = £17 per full ingestion
Integration Runtime: ~£0.18/hour × 48 hours = £8.64
Total: ~£26 per full sync

While £26 doesn't sound expensive, issues outside our control mean it's likely we'd need to do this several times a year. We'll also be running nightly incremental updates; although these process way smaller amounts of data, there will occasionally be bulk updates in the source system that require larger data volumes to be reingested.

Over a year, this all adds up. But more importantly, Synapse's orchestration overhead makes it unsuitable for high-volume, small operations regardless of cost.

Limited Resilience

Synapse pipelines support retry logic, but their error handling is coarse-grained. If one API call in a batch of 1,000 fails, the retry mechanism repeats the entire batch, not just the failed item. There's no built-in concept of a "poison message queue" for persistently failing records.

When we tested this with a deliberately flaky API endpoint, a single bad record caused the same batch to retry indefinitely, blocking progress on thousands of good records. This simply won't work for production ingestion where some records may have data quality issues.

Orchestration Overhead

Synapse pipelines introduce latency between activity transitions. In our testing, even with an empty pipeline activity, there's typically 3-5 seconds of overhead per activity execution. For 20,000 activities:

20,000 activities × 4 seconds overhead = 80,000 seconds = 22 hours

This overhead alone exceeds our entire time budget. The pipeline execution model is optimized for long-running data transformations, not for coordinating thousands of quick API calls.

Lack of Dynamic Scaling

Scaling Synapse pipeline execution requires manual configuration of integration runtime settings. You can't easily spin up hundreds of concurrent workers dynamically based on queue depth, then scale down to zero when work completes. This inflexibility makes it difficult to optimize both cost and performance.

The Queue-of-Work Pattern: A Better Approach

After ruling out Synapse pipelines, we needed a solution that could handle parallelism without orchestration overhead. The queue-of-work pattern emerged as the ideal approach, and it's surprisingly simple in concept: break the large job into thousands of small, independent work items, put them in a queue, and let multiple workers process them concurrently.

The pattern decouples work distribution from work execution, which turns out to be the key to solving all our challenges. Here's how it works:

Pattern Overview

┌─────────────────┐
│   Ingestor      │  1. Breaks large job into small work items
│   (Enqueuer)    │  2. Enqueues items to Azure Storage Queue
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Azure Storage  │  3. Durable, distributed queue
│     Queue       │  4. Supports automatic retry & poison handling
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│     Queue       │  5. Dequeues messages
│   Processor     │  6. Dispatches to work item processors
│   (Workers)     │  7. Processes API calls
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   Data Lake     │  8. Writes results
│   Storage       │
└─────────────────┘

Key Components

1. Work Items: Strongly-typed dataclasses that represent individual units of work (e.g., "fetch assets with IDs 1000-5999")

2. Work Queue: An abstraction over Azure Storage Queues that handles serialization, dequeuing, and poison message management. Using an abstraction makes testing simpler.

3. Work Item Processors: Decorated functions that execute the actual work (API calls, data transformation, persistence)

4. Queue Processor: A loop that dequeues messages, dispatches them to the appropriate processor, and handles errors

5. Work Item Dispatcher: A registry that maps work item types to their processors using decorator-based registration

How the Pattern Solves Our Challenges

1. Scalability Through Parallelism

The first benefit of this pattern became apparent when we broke down the ingestion workload. Instead of processing 1 million assets sequentially, we could divide them into manageable chunks:

# Breaking down 1 million assets into 200 independent work items
max_asset_id = 1_000_000
batch_size = 5_000

for start_id in range(0, max_asset_id, batch_size):
    work_queue.enqueue(
        IngestItemsByIdWorkItem(
            snapshot_time=timestamp,
            correlation_id=correlation_id,
            from_id=start_id,
            to_id=start_id + batch_size - 1
        )
    )

This creates 200 independent work items, each responsible for 5,000 assets. Why 5,000? We tested different batch sizes and found this offered the best balance - large enough to amortize queue overhead, small enough that a single failure doesn't waste too much work.

Now here's where it gets interesting. With these work items in the queue, we can run multiple queue processors simultaneously:

• Each processor dequeues and processes messages independently
• No coordination overhead between workers - they don't even know about each other
• Azure Storage Queues handle concurrent access automatically with optimistic concurrency
• Scale up the number of workers simply by deploying more instances

In our production environment, we run this as Azure Container Apps Jobs. We did some testing to find out what level of concurrency can be supported by the target API - sending too many concurrent requests would overload the API and cause failures to spike. As a result of this, during a full ingestion we typically scale to 12 concurrent workers. The math works out well:

20,000 API calls ÷ 12 workers = 1,666 API calls per worker
1,666 calls × 0.5 seconds average = 833 seconds = 14 minutes

In practice, we see full ingestion complete in under 2 hours due to API variability and processing overhead, but this is still a massive improvement over the roughly 48 hours sequential approach.

2. Fault Tolerance Through Retry and Poison Queues

One of the most challenging aspects of large-scale API ingestion is handling failures gracefully. APIs are unreliable - they timeout, return 500 errors, get rate-limited, or occasionally return malformed data. We needed the system to handle these failures without manual intervention.

Azure Storage Queues provide built-in retry semantics through visibility timeouts and dequeue counts. Here's how we leveraged them:

class AzureStorageWorkQueue:
    def __init__(self, queue, poison_queue, max_dequeues=5):
        self._queue = queue
        self._poison_queue = poison_queue
        self._max_dequeues = max_dequeues

    def dequeue(self):
        # Visibility timeout of 300 seconds (5 minutes)
        message = self._queue.receive_message(visibility_timeout=300)

        # Automatic poison message handling
        if message and message.dequeue_count > self._max_dequeues:
            # This message has failed 5+ times, move it to poison queue
            self._poison_queue.send_message(message.content)
            self._queue.delete_message(message)
            return None  # Skip this problematic message

        return DequeuedWorkItem(
            work_item=jsonpickle.decode(message.content),
            dequeued_message=message
        )

Note - to keep the code simpler here, I haven't shown any additional error handling logic around the calls to the storage API. Depending on the exact logic and control mechanisms you have an place, you should consider adding error handling and retry logic around API calls like queue.receive_message.

How this works in practice:

When a worker dequeues a message, it becomes invisible to other workers for 5 minutes. If the worker successfully processes it, the message is deleted. If the worker crashes or the API call fails, the message automatically becomes visible again after 5 minutes for another worker to retry.

Azure Storage Queues track how many times each message has been dequeued. After 5 failed attempts, we consider the message "poisoned". This is a standard term in message processing systems that refers to a message which has repeatedly failed to be dispatched or processed, likely due to bad data or a systematic issue. We don't want to waste resources by continually trying to process the message, but we don't want to lose it either so we move it to a separate "poison queue" for manual investigation.

It should be noted here that even if you've chosen a finite TTL for your message queue, you should ensure that messages on the poison queue never expire by setting their TTL to -1.

This approach has proven remarkably resilient, and it's unusual to see runs end with messages on the poison queue. We've added a check at the end of processing to raise an alert if this does happen so that the bad messages can be investigated. Options also exist inside Azure to monitor message queues for other scenarios - e.g. an excessive number of messages in either queue - and raise alerts.

This approach also aids recovery in the event of a catastrophic failure. For example, during one ingestion run, the source API had a 6-hour outage. This meant that the majority of the queue messages ended up on the poison queue. Once we realised what had happened, we were able to quickly move all of the messages from the poison queue back to the processing queue and rerun them.

3. Speed Through Asynchronous Processing

The queue processor implementation is deliberately simple, which turns out to be a performance advantage:

def process_queue(correlation_id, work_queue, logger, ...):
    while not queue_empty:
        message = work_queue.dequeue()
        if message is None:
            queue_empty = True
            continue

        try:
            # Dispatch to appropriate processor
            WorkItemDispatcher.dispatch_work_item(
                message.work_item,
                logger,
                **kwargs
            )
            # Success: remove from queue
            work_queue.remove(message)
        except Exception as e:
            # Failure: message stays in queue for retry
            logger.error(f"Error processing: {e}")

This simple loop runs on each worker container. There's no complex orchestration, no distributed locking, no coordination between workers. Each worker independently:

1. Dequeues a message (blocking call, returns when message available)
2. Processes it
3. Deletes the message on success
4. Repeats until the queue is empty

We measured the queue overhead itself - dequeue plus delete operations - at consistently under 50ms. This means for a typical 500ms API call, only 10% of the time is queue overhead. For slower API calls (2-3 seconds), the overhead becomes negligible.

The pattern also enables progressive completion. Unlike batch systems where you wait hours to see any results, data appears in the data lake as soon as each work item completes. This was particularly valuable during testing - we could verify the data pipeline was working correctly within minutes rather than waiting for a full ingestion to complete.

4. Clean Separation of Concerns

The dispatcher pattern keeps code maintainable:

# Define a work item type
@dataclass
class IngestItemsByIdWorkItem(WorkItem):
    from_id: int
    to_id: int

# Register its processor with a decorator
@work_item_processor(IngestItemsByIdWorkItem)
def process_items_by_id(work_item, logger, **kwargs):
    api_client = kwargs["api_client"]
    writer = kwargs["writer"]

    # Fetch data from API
    response = api_client.get_items(
        build_query_params(work_item.from_id, work_item.to_id)
    )

    # Persist to data lake
    writer.persist_assets(work_item.snapshot_time, response)

Benefits:

• Each processor focuses on one specific task
• Easy to add new work item types without modifying existing code
• Type safety through Python dataclasses
• Testable in isolation
• Clear dependency injection through kwargs

Important Considerations

Before implementing this pattern, there are several important factors to consider:

Data must be able to be partitioned: Since all the work is enumerated up front, you need a way of partitioning your data into a large number of small chunks. For our full ingestion, we chose ID ranges and for our incremental updates we chose time periods. This will be driven in part by the querying options supported by the API.

Idempotency is critical: Since messages can be retried automatically, your processors must be idempotent. If a worker crashes after persisting data but before deleting the message, another worker will process the same message again. In our implementation, we write data to the data lake with consistent file paths based on the work item parameters, so re-processing simply overwrites the existing file with identical data.

Message size limits: Azure Storage Queue messages are limited to 64 KB. Our work items are small (typically <1 KB when serialized), but if you need to pass large payloads, consider storing the data in blob storage and passing a reference in the message.

Visibility timeout tuning: The 5-minute visibility timeout works well for our API calls, but you'll need to adjust this based on your workload. Too short and messages might be retried while still being processed (duplicate work). Too long and failed messages won't be retried quickly enough.

Cost considerations: While Azure Storage Queue costs are minimal (~£0.00028 per 10,000 operations), container runtime costs can add up. With 50 workers running for 2 hours:

12 workers × 2 hours × £0.0001/vCPU-second × 0.5 vCPU × 3600 seconds
= approximately £4.32 per full ingestion

This is cheaper than the Synapse cost but provides much better performance and flexibility.

Avoid sensitive data in messages: Work items should contain only metadata (IDs, timestamps, pagination offsets), not actual sensitive data. The actual data from API responses goes directly to the data lake, not through the queue.

Real-World Implementation Example

Let's walk through a complete flow showing how these components work together for ingesting data:

Step 1: Enqueue Work (Main Orchestrator)

The first step runs in a dedicated "enqueuer" container that breaks the full ingestion job into work items:

class BronzeIngestor:
    def enqueue_full_ingestion(self, snapshot_time, correlation_id):
        # First, query the API to determine the scope of work
        # This makes a single API call to get the maximum asset ID
        max_asset_id = self._get_maximum_asset_id()

        # Break into batches of 5,000 assets each
        batch_size = 5_000
        current_id = 0

        while current_id < max_asset_id:
            self._work_queue.enqueue(
                IngestItemsByIdWorkItem(
                    snapshot_time=snapshot_time,
                    correlation_id=correlation_id,
                    from_id=current_id,
                    to_id=current_id + batch_size - 1
                )
            )
            current_id += batch_size

This enqueuing process typically completes in under a 30 seconds for 20,000 work items. The snapshot_time ensures all workers use the same timestamp for file paths, making the data lake files consistent. The correlation_id ties all work items to the same ingestion job for tracing.

Depending on how many items you end up needing to enqueue, and what else lives in your queue, you will also need to consider error and recovery scenarios here. What if you fail half way through enqueuing work items? Whilst the items themselves need to be idempotent, we don't want to put multiple of the same item on the queue if we can avoid it.

If you have one queue per process, then the simplest option will be to ensure the queue is empty before you start processing, and have a recovery process which clears messages down in case of error.

Step 2: Process Queue (Worker Containers)

Multiple worker containers run simultaneously, each executing the same code but processing different messages:

class BronzeIngestor:
    def process_queue(self, correlation_id):
        # Set up dependencies that processors will need
        processor_kwargs = {
            "api_client": self._api_client,
            "writer": self._bronze_writer
        }

        # This runs until the queue is empty
        # Each worker container runs this independently
        process_queue(
            correlation_id=correlation_id,
            work_queue=self._work_queue,
            logger=self._logger,
            get_processor_kwargs=lambda msg: processor_kwargs
        )

The process_queue function (shown in the "Speed" section above) is just a simple loop. When the queue is empty, the worker exits gracefully.

Step 3: Execute Work (Registered Processor)

The dispatcher routes each work item to its registered processor. The @work_item_processor decorator registers this function to handle IngestItemsByIdWorkItem instances:

@work_item_processor(IngestItemsByIdWorkItem)
def process_items_by_id(work_item, logger, **kwargs):
    api_client = kwargs["api_client"]
    writer = kwargs["writer"]

    # Each work item handles a range of IDs (e.g., 1000-5999)
    # But the API paginates responses, so we need an inner loop
    offset = 0
    limit = 100  # API returns 100 assets per page

    while True:
        # Make API call for one page of results within this ID range
        response = api_client.get_items(
            build_query_params(
                work_item.from_id,
                work_item.to_id,
                offset,
                limit
            )
        )

        assets = json.loads(response)["assets"]
        if not assets:
            break  # No more results, we're done with this work item

        # Write this page to the data lake
        # File path includes snapshot_time for consistency across workers
        writer.persist_assets(
            work_item.snapshot_time,
            work_item.from_id,
            work_item.to_id,
            offset,
            response
        )

        offset += limit

For a work item covering IDs 1000-5999 (5,000 assets), this typically makes 50 API calls (100 assets per page). The entire work item takes about 25 seconds to process, which fits comfortably within the 5-minute visibility timeout.

Step 4: Experimentation and tuning

Once you have a working implementation, then some experimentation is needed to validate and tune batch sizes and the amount of parallelism you can support.

When considering batch sizes, there are a variety of factors to take account of. Firstly, how many API requests will be required to ingest a single batch? As mentioned above, our 5,000 asset batches result in around 50 API calls.

If any one of them fails, they will all be retried. If you can't rely on the API to consistently ingest an entire batch of data without failure, you should tune your batch size to minimise the impact of this.

Also, how much data will the API return in a single batch? If you need to process an entire batch of data at once (even if it requires multiple API calls to retrieve), and your payload size is large, will you have enough memory to process it all?

How long will a single batch take to process? This has an effect on your queue visibility timeout; if your batch takes 5 minutes to process but your queue visibility timeout is 2 minutes, this means different instances of your queue processor will likely end up processing the same message.

When considering parallelism, this is mainly down to the API you're ingesting from. Depending on your deployment architecture (discussed below) the overall cost for the process may be similar regardless of whether you run 10, 20 or 50 processors in parallel. However, the API you're ingesting from might not be able to cope with this - at worst, you could end up crashing the API, or making it unresponsive for other users. In our scenario, the API we retrieve the data from is the same as that used by the front end application, so we needed to avoid this.

Clearly the two factors are linked. For example when evaluating the API we are ingesting from we discovered that:

• larger batch sizes are more efficient because the system behind the API caches the query, so retrieving subsequent pages for a batch is relatively fast compared to retrieving the first page.
• smaller page sizes are better, as the API struggled to serialize large pages of data quickly.
• the API could reliably cope with running 12 processors in parallel; more than that started to significantly impact production performance for other users. However, we also established that we got better results when running the ingestion process outside working hours.

This experimentation process needs to be done carefully. If you are experimenting on a production API, it's likely you need to work with the owners to ensure you don't end up rendering their API unsable. If you're working in a sandbox environment, you need to bear in mind that it will likely have a different allocation of compute resources to the production environment, so you may need to include levers to allow you to tune the process once you reach production.

Underpinning this experimentation is ensuring you have baked in observability so that you can evaluate the resulting telemetry to inform your decision - more on this later.

Deployment Architecture

In production, this pattern runs on Azure Container Apps, which provides the perfect hosting model for this workload.

Container Jobs

Azure Container Apps Jobs are designed for workloads that run to completion then exit - exactly what we need.

We have a single container job which is called with arguments to either run the enqueuing or processing. The job can be started multiple times to support parallelism.

Container size

ACA offers a variety of combinations of CPU and memory, which allows you to select an appropriate. The exact options depend on whether you're using a Dedicated plan or a Consumption one. We used a Consumption plan, which allows you to size your container from as small as 0.25 cores and 0.5 GB RAM up to 4 cores and 8 GB RAM (as of March 2026).

Since we're not doing a lot of processing, but we are processing reasonably large chunks of data, we chose the 0.5 core and 1 GB RAM option, and it's serving us well. As with most other things, choosing the right container size is a matter of doing some initial calculations and then experimenting until you achieve consistent performance and reliability.

Orchestration

Although we're running the enqueuing and processing logic in Azure Container apps, we're still orchestrating things via a Synapse pipeline. We chose this route for consistency; we're orchestrating all of the other processes in our data platform via Synapse pipelines and we didn't want to introduce other approaches.

We've created a pipeline that can trigger an Azure Container App Job, passing in the necessary parameters. This can run either as a fire-and-forget process, or it can poll for job completion. We use the former for the full ingestion, and the latter for the incremental ingestion as this allows us to immediately trigger data processing once ingestion is complete.

Cost Optimization

Container Apps Jobs only consume resources while running:

• Enqueue job: Runs for ~1 minute, costs negligible
• Worker jobs: Run for ~2 hours during full ingestion
• Zero cost when idle - no minimum running instances required

This is significantly more cost-effective than keeping functions warm or maintaining always-on compute resources.

Scaling Strategy

While Azure Container Apps supports automatic queue-based scaling using KEDA, we've found that starting with a fixed number of workers (12) works well for predictable workloads. For unpredictable workloads, especially when the target API can support a higher rate of requests than ours, you could configure scaling rules based on queue depth:

Queue depth > 1000 messages → Scale to 50 workers
Queue depth < 100 messages → Scale to 10 workers
Queue empty → Scale to 0

Observability

OpenTelemetry tracing provides end-to-end visibility. Each work item creates its own trace span, making it easy to:

• Identify slow API endpoints
• Track which work items failed and why
• Identify failure patterns
• Measure end-to-end ingestion duration
• Correlate all work items for a specific ingestion job using the correlation_id

Benefits Beyond the Original Requirements

While we initially focused on solving the core challenges of scale, fault tolerance, and speed, the pattern has delivered several unexpected benefits:

Incremental ingestion came for free: Once the framework was in place, adding incremental ingestion was trivial. We created a new work item type (IngestItemsModifiedSinceWorkItem) that queries for recently modified assets instead of ID ranges. The dispatcher, queue processing, and retry logic all work identically.

Debugging became significantly easier: When something goes wrong, the poison queue contains the exact work item that failed, complete with all parameters. We can inspect it, fix the underlying issue (often a data quality problem in the source system), then manually re-queue it without reprocessing thousands of successful items. This has saved hours of debugging time.

Testing improved dramatically: Previously, testing the full ingestion pipeline required running against the actual API or building complex mocks. Now we can test individual processors in isolation by constructing work items with test data. Integration tests can enqueue a few work items and verify the results without processing the entire dataset.

Cost visibility is excellent: Azure Storage Queue operations cost approximately £0.28 per million operations. For our 20,000-message ingestion, that's less than £0.01 in queue costs. The predictable, pay-per-use model makes capacity planning straightforward.

Ability to modify the structure of the written data: Data is received from our source system arrives as partially formatted JSON. This can cause issues with some processing libraries, such as Polars. We ended up rewriting the retrieved data using the ndjson format, which Polars can process with no issues.

When to Use This Pattern

The queue-of-work pattern is ideal when:

• You need to make thousands or millions of API calls
• Work can be divided into independent, idempotent units
• Fault tolerance is critical (some items may fail, but others should continue)
• You want to scale horizontally by adding more workers
• Processing time per item varies significantly
• You need to prioritize certain types of work

It may be overkill for:

• Small-scale operations (< 100 items)
• Tightly coupled sequential processing
• Real-time streaming (consider event-driven architectures instead)
• Simple ETL where Synapse pipelines or Data Factory are sufficient

Expansion to a second data source

We've since extended the pattern to a second data source in the same solution, Amazon Elasticsearch. This required us to go through the same process of evaluation and experimentation again, and brought with it a new consideration.

This time, the main benefit of the pattern is not in error handling, as the Elasticsearch service is highly resiliant and less susceptible to transient errors. However, it is highly scalable so means we can use a much higher degree of parallelism than with our original service.

Finally, size of individual data items is relatively small, but the volume of items we are retrieving is high, as each item represents a user event in the system. Regardless, we found that retrieving a day's worth of data at a time worked well.

Because of the high degree of parallelism permitted, a full ingestion of several year's worth of data takes seconds rather than the hours it would likely take if using a Synapse pipeline with an HTTP connector.

However, we had an additional consideration for this source; an IP allow-list on the service. In Synapse pipelines, this is dealt with using a Self Hosted Integration Runtime (SHIR) to allow you to ingest data from a well-known IP address. Fortunately, we already had the necessary infrastructure in place in our Azure Container App Environment, but this is something to be wary of when introducing additional moving parts into a solution.

Conclusion

The queue-of-work pattern transformed what initially seemed like an intractable data ingestion problem into a manageable, scalable solution. By decoupling work distribution from execution through Azure Storage Queues, we achieved the performance and reliability goals that seemed out of reach with traditional orchestration approaches.

The pattern has run in production for since mid 2024, ingesting large volumes of records weekly from multiple source systems. It's proven robust enough that ingestion failures are now rare, and when they do occur, the poison queue pattern makes debugging straightforward.

Key takeaways from this implementation:

1. Plan for the worst case: In a perfect world, we could potentially have lived with the sequential ingestion, as we'd only need to do the full ingestion once and then keep everything in sync. However, there are plenty of error scenarios, and these are what you need to make sure your process can handle.
2. Simple is fast: The basic queue-and-worker model introduces minimal overhead (<50ms per operation) compared to complex orchestration systems
3. Built-in resilience is valuable: Azure Storage Queues' native retry semantics and dequeue counting eliminated the need for custom retry logic
4. Horizontal scaling works: Adding workers linearly improves throughput without coordination overhead
5. Progressive completion aids debugging: Seeing results immediately rather than waiting hours made development and testing significantly faster
6. Determining batch sizes and concurrency limits is an experimental process: It will likely take time to determine the limits of the API you are using and there will be a number of factors to take into account. If you have a sandbox environment for the API, it will likely have different characteristics to production. And even if you find the limits of concurrency that an API can support, the owners of that API will likely not want you to push it to the limit as this could negatively impact other users.

When this pattern works well:

This approach excels when you can break work into independent units that don't depend on each other's results. The API ingestion scenario is ideal - fetching assets with IDs 1000-5999 has no dependency on fetching assets 6000-11999. If your workload requires sequential processing or complex dependencies between tasks, this pattern may not be the best fit.

If you've made it this far, thanks for reading! If you've got questions about implementing this pattern or would like to discuss your specific use case, feel free to leave a comment below. And as a final reminder, sample code to go with this blog post can be found in GitHub.

I showed some demos during the talk, and there was a request in the chat to make the source available. You can find the demos here:

https://github.com/endjin/rx-ondotnetlive-demos-2026

AIS.NET, Rx.NET and WPF

The first demo (source code at src/WpfAis) presents a map showing the current position of ships around the Norwegian coast:

This uses a data source provided by the Norwegian government. They monitor AIS data broadcast by ships around their coast, and make these messages available on a public endpoint. Endjin provides an open source suite of libraries collectively called Ais.Net that can be used to process data of this kind. This example uses the Ais.Net NuGet package to process the raw messages. It also uses the Ais.Net.Receiver package to retrieve messages from the Norwegian government's service, and to present those through Rx.NET.

This example shows how Rx.NET can be used to process live data streams declaratively. In particular, this effectively performs a 'join' over two kinds of message. Ships broadcast their names and types in different messages from the ones they use to report their locations (because their name and type tends to change a lot less often than their location). But the UI wants to combine this information so that it can display the ship's name and type over its location icon in the map.

The demo's viewmodel expresses this declaratively in Rx.NET:

IObservable<IGroupedObservable<uint, IAisMessage>> byVessel =
    receiverHost.Messages.GroupBy(m => m.Mmsi);
var vesselNavigationWithNameStream =
    from perVesselMessage in byVessel
    let vesselNavigationUpdates = perVesselMessage.OfType<IVesselNavigation>()
    let vesselNames = perVesselMessage.OfType<IVesselName>()
    let shipTypes = perVesselMessage.OfType<IShipType>()
    let vesselLocationsWithNames = Observable.CombineLatest(vesselNavigationUpdates, vesselNames, shipTypes,
        (navigation, name, type) => (navigation, name, type))
    from vesselLocationAndName in vesselLocationsWithNames
    select (mmsi: perVesselMessage.Key, vesselLocationAndName.name, vesselLocationAndName.navigation, vesselLocationAndName.type);

This uses the C# query expression syntax to describe the processing we require. The Rx.NET library does all the actual work for us. (If you'd like more information about how this works, I've shown a version of this query before at this talk.)

System.Linq.Async, .NET 10, and System.Linq.AsyncEnumerable

During the show, I talked about how for many years, the de facto implementation of LINQ for IAsyncEnumerable<T> System.Linq.Async was not, despite how the name makes it look, an officially supported library. It has always lived in the Rx.NET repo, and when we at endjin took over maintenance of Rx.NET, that meant we also became responsible for LINQ to IAsyncEnumerable<T>! (See the announcement video for the old System.Linq.Async package for more information on the history behind this.)

With .NET 10, this is now finally built into the .NET runtime libraries. The System.Linq.AsyncEnumerable package (which is officially supported by Microsoft) is included as part of .NET 10.0 but is also available for use on older runtimes.

In the demo I showed how this creates a potential problem for projects already using System.Linq.Async. The project at src/SysLinqDemo targets .NET 8.0 and uses System.Linq.Async v6. If you upgrade the project to use .NET 10.0, you'll get errors about ambiguous definitions of the LINQ operators:

I showed that simply removing the reference to System.Linq.Async is the simplest way to resolve this. But I also discussed how you might not be able to do that because you might be using some other library that has a dependency on it. So I also showed how upgrading to the latest version of System.Linq.Async (v7) also fixes this problem.

Rx 7 and UI framework support

I discussed how our main goal with Rx 7 is to fix the bloat problems that have for many years afflicted applications that use Rx.NET in conjunction with self-contained deployment when targetting Windows. This table shows the impact this problem can have on a simple 'hello world' console app:

For framework-dependent deployment, in which the .NET runtime is presumed already to be available, adding Rx.NET has a relatively small impact. But for any of the other options, a reference to System.Reactive can double or even triple the size of the deployment! This happens when you target a Windows-specific TFM because System.Reactive ends up forcing your project to depend on both WPF and Windows Forms. You end up deploying a copy of both of those frameworks, which is what's taking up all the space here. The only way we can fix this is to split out UI framework support from the main System.Reactive package, putting these features into more specialized packages.

The effect of this change is that for the four deployment models shown, the impact of adding Rx becomes roughly 1.6MB, 1.6MB, unmeasureably small, and 300KB respectively.

To demonstrate what this will look like for developers, I showed a simple WPF application (in src/WpfWithRx) that uses Rx.NET v6, and which relies on its ObserveOnDispatcher method to ensure that Rx notifications are handled on a suitable thread for performing UI updates. I then upgraded the project to the preview of Rx 7 to show what developers will see when they do this:

This illustrates that they will now get a build failure on code that expects UI framework support to be in the main Rx.NET package. But it also shows that Rx.NET v7 has an analyzer that detects this, and explains exactly how to resolve the problem. We hope this will make the transition relatively smooth for projects affected by this breaking change. (We also maintain binary compatibility—although the WPF and Windows Forms features have been removed from the public-facing API, they are actually still present in the runtime binaries.)

We've done this by writing a custom analyzer. (This is in the Rx.NET/Source/src/System.Reactive.Analyzers folder.) This is an extra DLL in the System.Reactive NuGet package that gets loaded by the IDE (Visual Studio, VS Code, or Rider will all find it). Analyzers inspect source code to find problems, and make suggestions for how to fix them.

(Ideally we would have supplied a Code Fix as well as an analyzer. The .NET SDK's analyzer mechanisms allow analyzers to propose code changes, which show up as 'fixes' suggested by the IDE. However, there isn't currently a good way for a fix to suggest changes to the .csproj file, which is what's required in this case. It seems that the only supported API for making the changes the fix would need to make is the old EnvDTE automation APIs offered by Visual Studio. However, that isn't available on other IDEs, so it wouldn't work on VS Code or Rider. And even in Visual Studio, there isn't a supported way for a code fix to get hold of that API. This surprised us a little, because these IDEs do actually make suggestions for adding NuGet package references in some other situations, but as far as we can see, there's no way for our analyzer/code fix to trigger that.)

Now that we've got an analyzer DLL built into System.Reactive, it would also be possible to start adding other analyzer rules: perhaps we could spot problematic coding patterns. If you have any ideas for common Rx issues that you think an analyzer could detect, please suggest them in https://github.com/dotnet/reactive/issues