Blog

Modern laptops can now handle datasets up to a billion rows, yet 94% of query spending goes on big-data compute that isn't needed. DuckDB brings analytical SQL directly into your process.

This blog explores how to implement robust validation strategies within the medallion architecture using Pandera, helping you catch issues early and maintain clean, trustworthy data.

Records are primarily meant for representing data. They are usually immutable and allow you to copy, equate, and print, object properties.

C# 12.0 adds a new way to annotate parameters: ref readonly. This seems like it should mean exactly the same as the older in annotation. This post explains why this new syntax is useful.

Understand why label encoding is the preferred technique for encoding categorical data for analysis in Power BI over one-hot encoding.

Covariance and contravariance in C# generic interfaces, unpacked from first principles using implicit reference conversions — demystified with simple examples.

One-hot encoding and label encoding are two methods used to encode categorical data. Understand the specific advantages and disadvantages of these techniques.

Explore integrating icons, pictograms and images into Power BI in the optimal way to enhance the user experience and minimise effort required to build and maintain reports.

Once you've thoroughly tested your code against the local Spark service in your dev container, you'll want to run it in a real Spark cluster. This posts shows how to deploy such code to Microsoft Fabric.

Having seen earlier in the series how to configure a dev container to run Spark locally, this post shows how to write tests that use that local Spark service.

Configure a Docker-based dev container that runs Spark locally — shorten your inner dev loop and make automated tests practical with the right base image.

Running Spark locally in a dev container can significantly improve development feedback loops. This first article explains why, and the rest of the series will show how.

C# 12.0 provides a new, simpler syntax for initializing expressions. It typically generates the most efficient code possible, although as you'll see, it's useful to understand the choices it makes.

Power BI's adoption of TMDL improves the readability of the semantic model, enables version control and enhances collaboration and efficiency for developers.

Women of Silicon Roundabout is the UK's largest women in tech event. Day two topics included: green tech, burnout, and Python!

Women of Silicon Roundabout is the UK's largest women in tech event. Day one topics included: AI, career pathways, and generations of Women in Tech.

A new feature in C# 12.0 enables data types to define fixed-size arrays that don't require separate array objects on the heap. Learn how this is useful in performance-oriented and interop scenarios.

Fix the following issues: Errors converting large datasets to pandas, pandas for Spark is very slow, and pandas for Spark column reduction doesn't reduce data.

We benchmarked endjin's JSON Schema library on .NET 9.0 and saw large performance gains. There are even more gains to be had with new System.Text.Json features.

.NET 9.0 has shipped, and for the fourth year running, we benchmarked endjin's AIS.NET library and were very happy to see substantial performance gains, with no extra work required.

Extending carbon-optimised pipelines: choose between Azure regions at runtime, work around Wait activity limits, and adapt the pattern beyond the UK.

A portable Data Factory, Synapse, or Fabric pipeline that calls the Carbon Intensity API and waits for the greenest scheduling window — no custom code.

We have a wide range of computational mechanisms at our disposal, some of which emerged thanks to recent advances in AI. In this post, we look at the kinds of workloads that can take advantage of these.

C# 12.0's most prominent new feature is the primary constructor syntax. This post describes how it works, and looks at some pros and cons.

Translating carbon-optimised scheduling into a modern data pipeline architecture for Microsoft Fabric, Azure Synapse and Azure Data Factory.