Building API Reference Documentation From Code, Part 2: Under the Hood

By Matthew Adams Co-Founder 11th August 2026 · 8 min read

In Part 1, we described the overall shape of our API reference documentation pipeline - what goes in, what comes out, and how the build and CI fit together. In this post, we'll look at how the key pieces work under the hood.

Assembly inspection and cross-linking

The first thing the tool does is a pre-scan across all assemblies in a version to build a combined type URL map:

Dictionary<string, string> combinedTypeUrlMap = new(StringComparer.Ordinal);
for (int i = 0; i < assemblyPaths.Count; i++)
{
    AssemblyInspector inspector = new(assemblyPaths[i]);
    Dictionary<string, string> partialMap = inspector.PreScanTypeUrls(resolvedBaseUrl);
    foreach (KeyValuePair<string, string> kvp in partialMap)
    {
        combinedTypeUrlMap[kvp.Key] = kvp.Value;
    }
}

This map is shared with the XML doc parser so that <see cref="T:Corvus.Text.Json.JsonElement"/> references in the Patch library resolve to the correct page URL in the core library. Without this pre-scan, cross-assembly links would be broken. The tool wouldn't know that JsonElement lives in a different assembly's page hierarchy.

The tool also builds a reverse map of interface implementations. For each interface type, it collects every concrete type that implements it:

foreach (TypeInfo typeInfo in allTypes)
{
    foreach ((string displayName, string? fullName) in typeInfo.InterfacesWithFullNames)
    {
        if (fullName is not null
            && typesByFullName.TryGetValue(fullName, out TypeInfo? ifaceInfo)
            && ifaceInfo.Kind == "interface")
        {
            ifaceInfo.ImplementedBy.Add((typeInfo.Name, typeInfo.FullName));
        }
    }
}

This means the page for IJsonElement<T> lists every generated type that implements it. That requires whole-solution analysis, not just single-assembly reflection.

Source links from PDB metadata

Every generated type page includes a "View Source" link pointing at the exact file and line on GitHub. Rather than guessing file paths from naming conventions, the tool reads Portable PDB metadata directly.

The SourceLinkResolver opens both the PDB and the compiled assembly, then builds a URL map in seven steps.

Step 1: Parse SourceLink JSON from the PDB

SourceLink embeds a JSON document in the PDB that maps local build paths to repository URLs. The resolver reads this from the module-level custom debug information using the well-known GUID CC110556-A091-4D38-9FEC-25AB9A351A6A:

foreach (CustomDebugInformationHandle cdiHandle in _pdbReader.GetCustomDebugInformation(moduleHandle))
{
    CustomDebugInformation cdi = _pdbReader.GetCustomDebugInformation(cdiHandle);
    if (_pdbReader.GetGuid(cdi.Kind) != SourceLinkGuid) { continue; }

    byte[] blob = _pdbReader.GetBlobBytes(cdi.Value);
    string json = System.Text.Encoding.UTF8.GetString(blob);
    // json = {"documents":{"D:\\source\\corvus-dotnet\\Corvus.JsonSchema\\*":
    //   "https://raw.githubusercontent.com/corvus-dotnet/Corvus.JsonSchema/COMMIT/*"}}

    using JsonDocument doc = JsonDocument.Parse(json);
    foreach (JsonProperty prop in doc.RootElement.GetProperty("documents").EnumerateObject())
    {
        string localPattern = prop.Name.Replace('\\', '/');   // normalise to forward slashes
        string urlPattern = prop.Value.GetString() ?? "";
        // Strip trailing '*' wildcards to get prefix pairs
        _sourceLinkMappings.Add((localPattern[..^1], urlPattern[..^1]));
    }
}

This gives us a list of (localPathPrefix, urlTemplate) pairs. Later, when we know a method lives in D:\source\corvus-dotnet\Corvus.JsonSchema\src\Corvus.Text.Json\Corvus\Text\Json\Document\JsonElement.cs, we can match against these prefixes and substitute the relative path into the URL template.

Step 2: Load embedded source

Modern .NET builds with <EmbedAllSources>true</EmbedAllSources> store the full source text of every file inside the PDB, either raw or Deflate-compressed. The resolver loads these into an in-memory cache:

byte[] blob = _pdbReader.GetBlobBytes(cdi.Value);
int uncompressedSize = BitConverter.ToInt32(blob, 0);

if (uncompressedSize == 0)
{
    sourceText = System.Text.Encoding.UTF8.GetString(blob, 4, blob.Length - 4);
}
else
{
    using var compressed = new MemoryStream(blob, 4, blob.Length - 4);
    using var deflate = new DeflateStream(compressed, CompressionMode.Decompress);
    using var reader = new StreamReader(deflate, System.Text.Encoding.UTF8);
    sourceText = reader.ReadToEnd();
}

_embeddedSourceCache[path] = sourceText.Split('\n');

This cache is critical for steps 6 and 7. It lets us find exact declaration lines for types and members that have no compiled method body.

Step 3: Map PE metadata tokens to type names

The resolver walks the PE metadata (the compiled assembly's type system) to build a lookup from metadata tokens to fully-qualified type names. This is what connects the PDB's debug information (which references methods by token) back to the type names we use in the documentation model.

Step 4: Walk method debug information for member URLs

This is the core loop. For every method in the PDB, the resolver reads its sequence points - the compiler-generated mapping from IL offsets to source locations - and extracts the first non-hidden line:

foreach (MethodDebugInformationHandle mdiHandle in _pdbReader.MethodDebugInformation)
{
    MethodDebugInformation mdi = _pdbReader.GetMethodDebugInformation(mdiHandle);
    // ...
    int firstLine = int.MaxValue;
    foreach (SequencePoint sp in mdi.GetSequencePoints())
    {
        if (!sp.IsHidden && sp.StartLine < firstLine)
        {
            firstLine = sp.StartLine;
        }
    }
    // Map back to the MethodDefinition in the PE, then to its declaring type
    MethodDefinitionHandle methodHandle = MetadataTokens.MethodDefinitionHandle(
        MetadataTokens.GetRowNumber(mdiHandle));
    MethodDefinition methodDef = _peMetadata.GetMethodDefinition(methodHandle);
    // ...
    string memberKey = $"{typeFullName}.{methodName}";
    string? memberUrl = BuildSourceUrl(filePath, firstLine);
    _sourceUrls[memberKey] = memberUrl;
}

After this step, every concrete method, property getter, and constructor has a URL. But interfaces, abstract members, enums, and delegates have no compiled method body. Their sequence points don't exist in the PDB.

Steps 5–6: Fill gaps for types and bodyless members

For types that had methods in step 4, the resolver already knows which source file(s) they live in. It selects a primary file (preferring BigNumber.cs over BigNumber.Parse.cs, for example), then scans the embedded source for the type declaration keyword:

foreach (string keyword in new[] { "class", "struct", "interface", "enum", "record", "delegate" })
{
    int kwIdx = line.IndexOf(keyword, StringComparison.Ordinal);
    // verify it's a word boundary, then look for the type name after the keyword
    int nameIdx = line.IndexOf(shortTypeName, afterKeyword, StringComparison.Ordinal);
    if (nameIdx >= 0) { return lineNumber; }
}

For types without any method debug info (interfaces with only inherited members, empty marker interfaces), the resolver falls back to the PDB's TypeDefinitionDocuments custom debug info - a Roslyn-specific extension that maps type tokens directly to source documents.

Step 7: Resolve abstract and interface members

Interface methods and abstract methods have no IL body, so they have no sequence points. The resolver handles these by scanning the embedded source for the member name:

foreach (MethodDefinitionHandle methodHandle in typeDef.GetMethods())
{
    // Skip if already resolved from sequence points
    if (_sourceUrls.ContainsKey(memberKey)) { continue; }

    // For properties, strip get_/set_ prefix to find the declaration
    string scanName = methodName.StartsWith("get_") ? methodName[4..] : methodName;
    int memberLine = FindMemberDeclarationLine(typeFile, scanName);
    if (memberLine > 0)
    {
        _sourceUrls[memberKey] = BuildSourceUrl(typeFile, memberLine);
    }
}

URL construction

Finally, BuildSourceUrl matches the local file path against the SourceLink mappings from step 1 and converts the raw.githubusercontent.com URL (which includes a commit SHA) into a browsable github.com/blob/main/ URL:

// raw.githubusercontent.com/owner/repo/COMMITSHA/path/to/file.cs
// → github.com/owner/repo/blob/main/path/to/file.cs#L42
string browsableUrl = ConvertToGitHubBlobUrl(rawUrl);
return $"{browsableUrl}#L{lineNumber}";

The end result is a Dictionary<string, string> with entries like Corvus.Text.Json.JsonElement → https://github.com/corvus-dotnet/Corvus.JsonSchema/blob/main/src/Corvus.Text.Json/Corvus/Text/Json/Document/JsonElement.cs#L24, covering types, methods, properties, constructors, and interface members. It contains about 8,000 entries across all 16 libraries.

TFM availability badges

Corvus.Text.Json targets net9.0, net10.0, netstandard2.0, and netstandard2.1. Some types and members only exist on newer frameworks - for example, Span<T>-based overloads aren't available on netstandard2.0.

The tool scans the netstandard2.0 and netstandard2.1 builds of each library, collecting a HashSet<string> of member keys present in each:

HashSet<string> partialKeys = AssemblyInspector.ScanMemberKeys(ns20Path);

Any type or member missing from a TFM's set gets flagged. The generated pages render this as availability information, so a user targeting netstandard2.0 can see at a glance which parts of the API they can use.

Hand-authored enrichments

Generated documentation from XML doc comments gives you accurate signatures and parameter descriptions. It doesn't give you the "here's how you'd actually use this" context that makes documentation valuable.

We solve this with two directories of hand-written markdown that the generator merges into generated pages:

• Namespace descriptions - a markdown file per namespace that appears at the top of the namespace page. For example, Corvus.Text.Json.md provides a prose overview with links to key types.

• Type examples - a markdown file per type that appears on the generated type page. For example, corvus-text-json-jsonelement.md demonstrates property access patterns, zero-allocation string comparison, and structural equality.

The key design decision: these files live in separate directories that the generator never deletes. When the tool regenerates all 3,300 V5 API pages, the 21 namespace descriptions and 25 type examples survive untouched. The generator loads them by slug-based convention, so no explicit registration is needed.

For namespace descriptions, the MarkdownGenerator checks for a file named {Namespace}.md in the descriptions directory and prepends its content to the namespace page:

if (namespaceDescriptionsDir is not null)
{
    string descPath = Path.Combine(namespaceDescriptionsDir, nsInfo.Name + ".md");
    if (File.Exists(descPath))
    {
        sb.Append(File.ReadAllText(descPath).TrimEnd());
    }
}

For type examples, it looks up a file named {nsSlug}-{typeSlug}.md and merges it into the type page - the same slug convention used for generated filenames:

private string? LoadExampleMarkdown(string slug)
{
    if (typeExamplesDir is null) { return null; }
    string examplePath = Path.Combine(typeExamplesDir, slug + ".md");
    if (File.Exists(examplePath))
    {
        return File.ReadAllText(examplePath).TrimEnd();
    }
    return null;
}

The same mechanism works at the member level. If you create a file matching a member's slug (e.g., corvus-text-json-jsonelement.parse.md), it appears on that member's detail page. This means enrichment can be as coarse-grained (one example per type) or fine-grained (per method overload group) as you need.

This separation is what makes enrichment sustainable. The alternative is to edit generated files directly. That means your hand-written content is destroyed every time the generator runs.

Version switcher

Both engine versions produce independent page trees (/api/v5/ and /api/v4/), each with their own sidebar, search index, and page hierarchy. The API landing pages include a version switcher so users can jump between the V5 and V4 reference. The build script passes version metadata when invoking the tool:

& dotnet run --project $toolProject -c Release -- `
    @v5ToolArgs `
    --output $v5ApiContentDir `
    --taxonomy-output $v5ApiTaxonomyDir `
    --api-views-dir $v5ApiViewsDir `
    --shared-views-dir $sharedViewsDir `
    --api-base-url /api/v5 `
    --version-label "V5 Engine" `
    --alt-version-label "V4 Engine" `
    --alt-version-url "/api/v4/index.html"

This is especially important during the V4 → V5 migration period, when users frequently need to compare the same concept across both APIs.

Per-version search

Each version gets its own Lunr search index, generated as a JSON file. The SearchIndexGenerator walks every namespace, type, and member, building a search entry for each:

foreach (TypeInfo type in kvp.Value.Types)
{
    // Build keywords from the type
    List<string> keywords = [type.Name, type.Kind, ns];
    keywords.AddRange(type.GenericParameters);

    // Build body from all member summaries
    StringBuilder body = new();
    if (!string.IsNullOrEmpty(type.Documentation?.Summary))
    {
        body.AppendLine(type.Documentation!.Summary);
    }
    foreach (MemberInfo method in type.Methods)
    {
        if (!string.IsNullOrEmpty(method.Documentation?.Summary))
        {
            body.AppendLine($"{method.Name}: {method.Documentation!.Summary}");
        }
    }

    entries.Add(new SearchEntry
    {
        Url = typeUrl,
        Title = type.Name,
        Description = type.Documentation?.Summary ?? string.Empty,
        Keywords = string.Join(" ", keywords),
        Body = body.ToString().Trim(),
    });
}

Each entry combines the type or member name, its XML doc summary, its kind (struct, interface, etc.), generic parameter names, and all child member summaries into a single searchable body. This means searching for "parse" finds not just types named Parse, but any type whose methods include Parse in their documentation.

Individual member pages get their own entries too. Constructors, properties, method overload groups, and operators each produce a search result with a direct URL to the member detail page.

The search UI on the API landing page loads the index for the version the user is currently viewing. A global site-wide search index (built separately by a Node.js tool in a later pipeline step) covers non-API content like tutorials and recipes.

The pattern

The specific tools we chose are Vellum for static site generation, Lunr for search, and lychee for link checking. These are our choices for our particular situation. The pattern underneath them is more general:

1. Generate from binaries, not source: assembly reflection gives you the actual public API surface, including generic constraints, interface implementations, and inheritance that aren't always obvious from source. XML doc comments give you the human descriptions. PDBs give you source locations.

2. Separate generated and hand-authored content: keep enrichments in a directory the generator never writes to. Load them by naming convention. This makes regeneration safe and enrichment sustainable.

3. Pre-scan for cross-linking: when documenting multiple assemblies, build a combined type URL map before generating any pages. Otherwise cross-assembly references break.

4. Scan multiple TFM builds: if your library multi-targets, scan each TFM's compiled assembly to determine per-member availability. Users targeting older frameworks need to know what they can and can't use.

5. Validate links as part of the build: broken internal links are the first thing that rots when APIs change. A link checker in CI catches them before they reach users.

6. Integrate with your CI pipeline: documentation that isn't built in CI is documentation that drifts.

If you maintain a .NET library with more than a handful of public types, the investment in a custom documentation pipeline pays for itself the first time an API change silently breaks a link or renders a code sample invalid. The code is open source. Feel free to explore the XmlDocToMarkdown tool, the build pipeline, and the enrichment directories to see how the pieces fit together.

FAQs