Editions

.NET Aspire can create new local instances of services such as SQL Server from scratch each time you run. While this guarantees repeatability and isolation, it can be time consuming, so this post explores alternatives.

.NET Aspire can create all of the resources an integration test required, but the test will often need direct access to the same SQL database as the code under test. This post shows how to do that.

.NET Aspire can create local services such as a SQL Server to stand in for cloud resources under local development and testing. This post shows how to ensure such a database is suitably initialized.

This post describes an approach to copy files and data from SharePoint into Azure using Synapse Pipelines.

The interactive notebook shared in this post defines the process of granting Service Principals (inc. Synapse managed identities) access to SharePoint sites.

The Medallion Architecture consists of three data tiers: Bronze (raw), Silver (clean), and Gold (projected). Data moves through these three tiers and becomes more opinionated at each stage.

Liam joined endjin as part of the Software Engineering Apprenticeship 2021 cohort. In this post he looks back on his time at endjin before moving on.

Eli joined endjin as part of the Software Engineering Apprenticeship 2021 cohort. In this post she reflects on her first two years.

As Charlotte's placement comes to an end, she reflects on her year at endjin, highlighting her experiences to take back to University with her

Records are primarily meant for representing data. They are usually immutable and allow you to copy, equate, and print, object properties.

Intelligently scheduling cloud data pipelines based on carbon impact can optimize both environmental sustainability and operational efficiency.

Intelligently scheduling cloud data pipelines based on carbon impact can optimize both environmental sustainability and operational efficiency.

In this series of posts, we look at all the places where you can integrate Power Query as part of your data solutions. Here we look at Power BI.

In this series of posts, we look at all the places where you can integrate Power Query as part of your data solutions. Here we look at Microsoft 365.

Learn to configure AAD or MSA 2FA profiles for UI automation testing with Time-based One-Time Passwords.

This post describes an approach to copy files and data from SharePoint into Azure using Synapse Pipelines.

The interactive notebook shared in this post defines the process of granting Service Principals (inc. Synapse managed identities) access to SharePoint sites.

With new technologies - such as Delta Lake and other open table formats - there have been huge improvements the performance of Data Lakehouses. But what is Delta Lake and how does it work?

This post describes an approach to copy files and data from SharePoint into Azure using Synapse Pipelines.

The interactive notebook shared in this post defines the process of granting Service Principals (inc. Synapse managed identities) access to SharePoint sites.

Running Spark locally in a dev container can significantly improve development feedback loops. This first article explains why, and the rest of the series will show how.

What exactly is a Data Lakehouse? This blog gives a general introduction to their history, functionality, and what they might mean for you!

Serverless data architectures enable leaner data insights and operations. How do you reap the rewards while avoiding the potential pitfalls?

New Azure Synapse Analytics service offers SQL Serverless for on-demand data lake queries. We tested its potential as a Data Lake Analytics replacement.

Explore DuckLake's advanced capabilities including built-in encryption, sophisticated conflict resolution, and the strategic implications for future data architecture. Understand how DuckLake enables new business models and positions itself against established lakehouse formats.

Get hands-on with DuckLake through a comprehensive tutorial covering installation, basic operations, file organization, snapshots, and time travel functionality. Learn how DuckLake's database-backed metadata management works in practice.

DuckDB Labs introduces DuckLake, a revolutionary approach to lakehouse architecture that solves fundamental problems with existing formats by bringing database principles back to data lake metadata management.

This post describes an approach to copy files and data from SharePoint into Azure using Synapse Pipelines.

The interactive notebook shared in this post defines the process of granting Service Principals (inc. Synapse managed identities) access to SharePoint sites.

Intelligently scheduling cloud data pipelines based on carbon impact can optimize both environmental sustainability and operational efficiency.

Liam joined endjin as part of the Software Engineering Apprenticeship 2021 cohort. In this post he looks back on his time at endjin before moving on.

Women of Silicon Roundabout is the UK's largest women in tech event. Day two topics included: green tech, burnout, and Python!

Women of Silicon Roundabout is the UK's largest women in tech event. Day one topics included: AI, career pathways, and generations of Women in Tech.

The Medallion Architecture consists of three data tiers: Bronze (raw), Silver (clean), and Gold (projected). Data moves through these three tiers and becomes more opinionated at each stage.

Understand why label encoding is the preferred technique for encoding categorical data for analysis in Power BI over one-hot encoding.

One-hot encoding and label encoding are two methods used to encode categorical data. Understand the specific advantages and disadvantages of these techniques.

Explore DuckLake's advanced capabilities including built-in encryption, sophisticated conflict resolution, and the strategic implications for future data architecture. Understand how DuckLake enables new business models and positions itself against established lakehouse formats.

Get hands-on with DuckLake through a comprehensive tutorial covering installation, basic operations, file organization, snapshots, and time travel functionality. Learn how DuckLake's database-backed metadata management works in practice.

DuckDB Labs introduces DuckLake, a revolutionary approach to lakehouse architecture that solves fundamental problems with existing formats by bringing database principles back to data lake metadata management.

This is follow guide to designing a mobile navigation in Power BI, covering form, icons, states, actions, with a view to enhancing report design & UI.

What exactly is a Data Lakehouse? This blog gives a general introduction to their history, functionality, and what they might mean for you!

Microsoft Fabric unifies data & analytics, building on Azure Synapse Analytics for improved data-level interoperability. Explore its offerings & pros/cons.

Explore how to implement source control in Databricks notebooks, promoting software engineering best practices.

Learn how to access multi-select choice column choice labels from Azure Synapse Link for Dataverse using PySpark or SQL.

Learn how to access the choice labels from Azure Synapse Link for Dataverse using T-SQL through SQL Serverless and by using Spark SQL in a Synapse Notebook.

Learn how to access the choice labels from Azure Synapse Link for Dataverse using PySpark.

Polyglot Notebooks' PowerShell support enhances IT Ops with robust, repeatable processes via 'executable documentation'.

Open-source software has become an essential part of many organisation's software supply chain, however, this poses challenges with license compliance and security assurance.

Implement Python data validation with Pandera & Great Expectations in this comparison of their features and use cases.

C# 11.0 expanded high-performance, low-allocation functionality. This post describes the importance of the added support for ref fields, and how the scoped keyword relates.

Architectural Decision Records (ADRs) capture context, options, decisions, and consequences. dotnet-adr is a .NET tool for managing ADRs.

As data and AI become the engine of business change, we need to learn the lessons of the past to avoid expensive failures.

Explore monetizing APIs with our guide. We offer strategies, videos, and code via Azure API Management to fast-track your business model.

Some of my thoughts inspired whilst helping out on the farm over the weekend. What is the future of work given the increasing presence of machines in our day to day lives? In which situations can AI deliver greatest value? How can we ease the stress of digital transformation on people who are impacted by it?

Serverless data architectures enable leaner data insights and operations. How do you reap the rewards while avoiding the potential pitfalls?

Some of my thoughts inspired whilst helping out on the farm over the weekend. What is the future of work given the increasing presence of machines in our day to day lives? In which situations can AI deliver greatest value? How can we ease the stress of digital transformation on people who are impacted by it?

Wishing Azure Synapse Analytics had support for SQL notebooks? Fear not, it's easy to take advantage rich interactive notebooks for SQL Pools and SQL on Demand.

Tracking down unexpected allocations in a high-performance .NET parsing library.

What exactly is a Data Lakehouse? This blog gives a general introduction to their history, functionality, and what they might mean for you!

This is a summary of the sessions I attended at SQLbits 2024 - Europe's largest expert led data conference. This year SQLBits was hosted at Farnborough IECC, Hampshire.

This is a summary of the sessions I attended at SQLbits 2023 in Newport Wales, which is Europe's largest expert led data conference.

Explore DuckLake's advanced capabilities including built-in encryption, sophisticated conflict resolution, and the strategic implications for future data architecture. Understand how DuckLake enables new business models and positions itself against established lakehouse formats.

Get hands-on with DuckLake through a comprehensive tutorial covering installation, basic operations, file organization, snapshots, and time travel functionality. Learn how DuckLake's database-backed metadata management works in practice.

DuckDB Labs introduces DuckLake, a revolutionary approach to lakehouse architecture that solves fundamental problems with existing formats by bringing database principles back to data lake metadata management.

We have a wide range of computational mechanisms at our disposal, some of which emerged thanks to recent advances in AI. In this post, we look at the kinds of workloads that can take advantage of these.

Thanks in part to recent advances in AI, we have a range of computational mechanisms at our disposal. However, certain universal truths apply to all of them.

Recent huge investment in AI has changed the modern computational landscape. Whatever the value of recent AI developments ultimately proves to be, we have some new hardware capabilities as a side effect. What else do these enable?

What exactly is a Data Lakehouse? This blog gives a general introduction to their history, functionality, and what they might mean for you!

Learn Reactive Programming with our free book, Introduction to Rx.NET 2nd Edition (2024), available in PDF, EPUB, online, and GitHub.

After a year of working on implementing the OpenChain specification, this blog takes you through the processes we created to track and manage our open-source licenses

Open-source software has become an essential part of many organisation's software supply chain, however, this poses challenges with license compliance and security assurance.

When implementing OpenChain, understanding the specification will help guide your organisation to having processes in place to review and manage open-source software

The key risks associated with open-source software, from whether you use it minimally, to using it throughout all your systems.

In Paul Waller's final, and posthumously published blog post, he takes you through a creative walk-through of the Titanic Power BI Report he created with Barry Smart.

This is follow guide to designing a mobile navigation in Power BI, covering form, icons, states, actions, with a view to enhancing report design & UI.

Understand why label encoding is the preferred technique for encoding categorical data for analysis in Power BI over one-hot encoding.

What exactly is a Data Lakehouse? This blog gives a general introduction to their history, functionality, and what they might mean for you!

This blog explores how to implement robust validation strategies within the medallion architecture using Pandera, helping you catch issues early and maintain clean, trustworthy data.

Learn how to configure VS Code to enable stepping into external code when debugging a Python Behave test

No-code/Low-code democratizes software development with little to no coding skills needed. But how do you evaluate if software DIY is the right choice for you?

Open-source software has become an essential part of many organisation's software supply chain, however, this poses challenges with license compliance and security assurance.

When implementing OpenChain, understanding the specification will help guide your organisation to having processes in place to review and manage open-source software

Once you've thoroughly tested your code against the local Spark service in your dev container, you'll want to run it in a real Spark cluster. This posts shows how to deploy such code to Microsoft Fabric.

Having seen earlier in the series how to configure a dev container to run Spark locally, this post shows how to write tests that use that local Spark service.

See how to configure a dev container to run Spark locally, to improve development feedback loops.

Explore monetizing APIs with our guide. We offer strategies, videos, and code via Azure API Management to fast-track your business model.

Microsoft recently shot a video interviewing endjin co-founder, Howard van Rooijen, and Director of Engineering, James Broome, about how Microsoft has helped endjin grow over the past decade. This posts the top 10 ways in which Microsoft helped - from providing access to valuable software and services, to opening up sales channels, to helping to navigate the minefield of UK Financial Services regulations around cloud adoption.

In this post we discuss the characteristics of a great FinTech startup, and the importance of the API Economy to innovation in Financial Services.

This guide captures the essential points that leaders should consider when setting up a new data team.

Our experience shows that the the most successful data projects rely heavily on building a multi-disciplinary team.

As data and AI become the engine of business change, we need to learn the lessons of the past to avoid expensive failures.

In Paul Waller's final, and posthumously published blog post, he takes you through a creative walk-through of the Titanic Power BI Report he created with Barry Smart.

This is follow guide to designing a mobile navigation in Power BI, covering form, icons, states, actions, with a view to enhancing report design & UI.

Explore integrating icons, pictograms and images into Power BI in the optimal way to enhance the user experience and minimise effort required to build and maintain reports.

In Paul Waller's final, and posthumously published blog post, he takes you through a creative walk-through of the Titanic Power BI Report he created with Barry Smart.

This is follow guide to designing a mobile navigation in Power BI, covering form, icons, states, actions, with a view to enhancing report design & UI.

Explore integrating icons, pictograms and images into Power BI in the optimal way to enhance the user experience and minimise effort required to build and maintain reports.