Blog

Configure a Docker-based dev container that runs Spark locally — shorten your inner dev loop and make automated tests practical with the right base image.

Running Spark locally in a dev container can significantly improve development feedback loops. This first article explains why, and the rest of the series will show how.

C# 12.0 provides a new, simpler syntax for initializing expressions. It typically generates the most efficient code possible, although as you'll see, it's useful to understand the choices it makes.

Power BI's adoption of TMDL improves the readability of the semantic model, enables version control and enhances collaboration and efficiency for developers.

Women of Silicon Roundabout is the UK's largest women in tech event. Day two topics included: green tech, burnout, and Python!

Women of Silicon Roundabout is the UK's largest women in tech event. Day one topics included: AI, career pathways, and generations of Women in Tech.

A new feature in C# 12.0 enables data types to define fixed-size arrays that don't require separate array objects on the heap. Learn how this is useful in performance-oriented and interop scenarios.

Fix the following issues: Errors converting large datasets to pandas, pandas for Spark is very slow, and pandas for Spark column reduction doesn't reduce data.

We benchmarked endjin's JSON Schema library on .NET 9.0 and saw large performance gains. There are even more gains to be had with new System.Text.Json features.

.NET 9.0 has shipped, and for the fourth year running, we benchmarked endjin's AIS.NET library and were very happy to see substantial performance gains, with no extra work required.

Extending carbon-optimised pipelines: choose between Azure regions at runtime, work around Wait activity limits, and adapt the pattern beyond the UK.

A portable Data Factory, Synapse, or Fabric pipeline that calls the Carbon Intensity API and waits for the greenest scheduling window — no custom code.

We have a wide range of computational mechanisms at our disposal, some of which emerged thanks to recent advances in AI. In this post, we look at the kinds of workloads that can take advantage of these.

C# 12.0's most prominent new feature is the primary constructor syntax. This post describes how it works, and looks at some pros and cons.

Translating carbon-optimised scheduling into a modern data pipeline architecture for Microsoft Fabric, Azure Synapse and Azure Data Factory.

Cloud data pipelines often have flexibility in when they run. Using the UK National Grid Carbon Intensity API, you can schedule them for the greenest window.

Thanks in part to recent advances in AI, we have a range of computational mechanisms at our disposal. However, certain universal truths apply to all of them.

Learn how to configure VS Code to enable stepping into external code when debugging a Python Behave test

C# 11.0 expanded high-performance, low-allocation functionality. This post describes the importance of the added support for ref fields, and how the scoped keyword relates.

AI's current hype cycle has driven massive hardware investment. Even if AI disappoints, those hardware capabilities remain. What else can we do with them?

Power BI's new PBIR format enhances collaboration, version control, and efficiency for developers. Learn key benefits and future implications.

Power BI Projects are a game changer for teams building reports; offering a source-control friendly format, CI/CD support, and the ability to edit in a code editor.

Spark doesn't always interpret JSON how we'd like. For example, if each key/value pair in a JSON object is conceptually one item, Spark won't give you a row per item by default. This article shows how to nudge Spark in the right direction.

This post examines .NET's native support for iterators: IEnumerator<T>, IEnumerable<T>, and IAsyncEnumerable<T>.

High-performance data teams start with C-suite sponsorship, align to organisational goals, adopt a product mindset, and position themselves as innovation engines rather than order takers.