Browse our archives by topic…
Editions
.NET Development
View all (310)
Rx.NET v6.0 Now Available
For the first time since 2020, a new release of Rx.NET is available, supporting .NET 6 and .NET 7.

C# 11.0 new features: Span<char> pattern matching
C# 11.0 adds some new pattern matching features. In this post, Ian shows the new support for matching lists.

Using the Playwright C# SDK to automate 2FA authentication for AAD and MSA
Enabling 2FA is a must, but can often be perceived as a blocker for UI automation testing. Here's how to configure your AAD or MSA 2FA profile so that you can automate auth with a Time-based One-Time Password
Analytics
View all (125)
Microsoft Fabric: Announced
Microsoft Fabric extends the promise of Azure Synapse integration to all analytics workloads from the data engineer to the business knowledge worker. It brings together Power BI, Data Factory, and the Data Lake, on a new generation of the Synapse data infrastructure. Delivered as a unified SaaS offering, it aims to reduce cost and time to value, while enabling new "citizen data science" capabilities. Check out all the resources from the endjin team collated in this post.

What is OneLake?
OneLake is the core of Microsoft Fabric - it is essentially the storage account for all of your data utilized within Fabric, be that within Azure or from another cloud (through Shortcuts). As the name suggests, it is a single, logical data lake backing more or less all of your Fabric workloads. For fans of analogies: you'll have probably heard Microsoft use the tag-line "OneLake is to data what OneDrive is to files/documents". Read this blog to dive deeper into OneLake and learn about why you should be excited.

Intro to Microsoft Fabric
Microsoft Fabric is a unified data & analytics platform that brings together, and improves upon, Microsoft's existing suite of data products. It can be thought of as the third generation of Microsoft data platforms, where first generation (e.g. HDInsight, SQL Data Warehouse) were somewhat isolated takes on traditional data products; second generation was Azure Synapse Analytics, which integrated platforms at a UX level but still felt a little disjointed at the data level; and now we have Microsoft Fabric which builds upon the Synapse "unification" vision, with a particular focus on enabling deep data-level interoperability. In this post we introduce its core offerings, as well as weigh up the pros and cons of this new product.
Apprenticeship
View all (55)
My Year in Industry so far
Charlotte is studying for a Bachelor of Engineering - BEng (Hons), Computer Science at the University of York. She was part of our 2021 internship cohort, and is spending her Year in Industry placement with endjin too. In this post she reflects on her first 6 months.

My first year as an apprentice software engineer at endjin
Jessica joined endjin as part of the Software Engineering Apprenticeship 2021 cohort. In this post, Jessica reflects on her first year.

Retrospecting on my first year at endjin
Liam joined endjin as part of the Software Engineering Apprenticeship 2021 cohort. In this post he looks back on his fist year and uses the post to introspect and look ahead to his second year.
Architecture
View all (64)
Using Cloud CI/CD in Zero Trust Environments
Performing certain deployment operations from a cloud-based CI/CD agent against resources that are only accessible via private networking are problematic. The often-cited solution is to use your own hosted agents that are also connected to the same private networks, however, this introduces additional costs and maintenance overhead. This post discusses an approach that combines the use of cloud-based CI/CD agents with 'just-in-time' allow-listing as an alternative to operating your own private agents.

Creating Pester Unit Tests in PowerShell
Why you should make writing unit tests a regular practice and how they will improve your code. This blog explains how to write unit tests in PowerShell using Pester.

Adventures in Dapr: Episode 5 - Debugging Containers with Visual Studio
If you're interested in the Distributed Application Runtime (Dapr) and how it can simplify migrating existing workloads to use cloud platform services, then this series might be for you. In this post, we use the container tools in Visual Studio to debug why a newly-containerised service is not working.
Automation
View all (4)
Using the Playwright C# SDK to automate 2FA authentication for AAD and MSA
Enabling 2FA is a must, but can often be perceived as a blocker for UI automation testing. Here's how to configure your AAD or MSA 2FA profile so that you can automate auth with a Time-based One-Time Password

How to setup Python, PyEnv & Poetry on Windows
Using multiple versions of Python on Windows can be a frustrating experience, especially if you've experienced how smooth the same workflow is on MacOS and Linux/WSL. This post provides a script and guide to getting setup to use Python virtual environments and Poetry, a popular dependency management tool.

Adventures in GitHub Actions: Episode 1 - Goodbye Azure DevOps, Hello GitHub Actions
This series will follow our ongoing journey of migrating the CI/CD pipelines for our many Open Source repositories, from Azure DevOps to GitHub Actions.
Azure
View all (179)
Microsoft Fabric: Announced
Microsoft Fabric extends the promise of Azure Synapse integration to all analytics workloads from the data engineer to the business knowledge worker. It brings together Power BI, Data Factory, and the Data Lake, on a new generation of the Synapse data infrastructure. Delivered as a unified SaaS offering, it aims to reduce cost and time to value, while enabling new "citizen data science" capabilities. Check out all the resources from the endjin team collated in this post.

What is OneLake?
OneLake is the core of Microsoft Fabric - it is essentially the storage account for all of your data utilized within Fabric, be that within Azure or from another cloud (through Shortcuts). As the name suggests, it is a single, logical data lake backing more or less all of your Fabric workloads. For fans of analogies: you'll have probably heard Microsoft use the tag-line "OneLake is to data what OneDrive is to files/documents". Read this blog to dive deeper into OneLake and learn about why you should be excited.

Intro to Microsoft Fabric
Microsoft Fabric is a unified data & analytics platform that brings together, and improves upon, Microsoft's existing suite of data products. It can be thought of as the third generation of Microsoft data platforms, where first generation (e.g. HDInsight, SQL Data Warehouse) were somewhat isolated takes on traditional data products; second generation was Azure Synapse Analytics, which integrated platforms at a UX level but still felt a little disjointed at the data level; and now we have Microsoft Fabric which builds upon the Synapse "unification" vision, with a particular focus on enabling deep data-level interoperability. In this post we introduce its core offerings, as well as weigh up the pros and cons of this new product.
Azure Synapse Analytics
View all (29)
Microsoft Fabric: Announced
Microsoft Fabric extends the promise of Azure Synapse integration to all analytics workloads from the data engineer to the business knowledge worker. It brings together Power BI, Data Factory, and the Data Lake, on a new generation of the Synapse data infrastructure. Delivered as a unified SaaS offering, it aims to reduce cost and time to value, while enabling new "citizen data science" capabilities. Check out all the resources from the endjin team collated in this post.

What is OneLake?
OneLake is the core of Microsoft Fabric - it is essentially the storage account for all of your data utilized within Fabric, be that within Azure or from another cloud (through Shortcuts). As the name suggests, it is a single, logical data lake backing more or less all of your Fabric workloads. For fans of analogies: you'll have probably heard Microsoft use the tag-line "OneLake is to data what OneDrive is to files/documents". Read this blog to dive deeper into OneLake and learn about why you should be excited.

Azure Synapse Analytics versus Microsoft Fabric: A Side by Side Comparison
In this post we compare Azure Synapse Analytics with Microsoft Fabric to understand how features map from Synapse to Fabric.
Big Compute
View all (27)
Azure Synapse Analytics: How serverless is replacing the data warehouse
Serverless data architectures enable leaner data insights and operations. How do you reap the rewards while avoiding the potential pitfalls?

Benchmarking Azure Synapse Analytics - SQL Serverless, using Polyglot Notebooks
There is a new service in town that promises to transform the way you query the contents of your data lake. Azure Synapse Analytics comes with a new offering called SQL Serverless allowing you to query your data on-demand with no need for pre-provisioned resources.When we heard about the new service we were keen to get involved, so for the last 10 months we've been working with the SQL Serverless product group to provide feedback on the service and to help ensure it meets our customers needs. During this time we've put it through it's paces by implementing a range of real-world use cases. We were particularly interested to see how it stacked up as a replacement for Data Lake Analytics, where to date there has been no clear and easy migration path.

Does Azure Synapse Analytics spell the end for Azure Databricks?
Have you or are you about to invest in Azure Databricks? If so, the new Spark offering in Azure Synapse Analytics is likely to have grabbed your attention and rightly so. Why is Microsoft putting yet another Spark offering on the table and what does it mean for you?
Big Data
View all (85)
Microsoft Fabric: Announced
Microsoft Fabric extends the promise of Azure Synapse integration to all analytics workloads from the data engineer to the business knowledge worker. It brings together Power BI, Data Factory, and the Data Lake, on a new generation of the Synapse data infrastructure. Delivered as a unified SaaS offering, it aims to reduce cost and time to value, while enabling new "citizen data science" capabilities. Check out all the resources from the endjin team collated in this post.

What is OneLake?
OneLake is the core of Microsoft Fabric - it is essentially the storage account for all of your data utilized within Fabric, be that within Azure or from another cloud (through Shortcuts). As the name suggests, it is a single, logical data lake backing more or less all of your Fabric workloads. For fans of analogies: you'll have probably heard Microsoft use the tag-line "OneLake is to data what OneDrive is to files/documents". Read this blog to dive deeper into OneLake and learn about why you should be excited.

Intro to Microsoft Fabric
Microsoft Fabric is a unified data & analytics platform that brings together, and improves upon, Microsoft's existing suite of data products. It can be thought of as the third generation of Microsoft data platforms, where first generation (e.g. HDInsight, SQL Data Warehouse) were somewhat isolated takes on traditional data products; second generation was Azure Synapse Analytics, which integrated platforms at a UX level but still felt a little disjointed at the data level; and now we have Microsoft Fabric which builds upon the Synapse "unification" vision, with a particular focus on enabling deep data-level interoperability. In this post we introduce its core offerings, as well as weigh up the pros and cons of this new product.
Big DataServerless
View all (1)
Azure Synapse Analytics versus Microsoft Fabric: A Side by Side Comparison
In this post we compare Azure Synapse Analytics with Microsoft Fabric to understand how features map from Synapse to Fabric.
Culture
View all (125)
Agile teams often get their planning and priorities wrong
Do you struggle to manage conflicting demands from multiple stakeholders, even when they have agreed about the relative priority of their requests? Apply a budgeting methodology to agile projects for better planning and communication with the team and wider stakeholders.

Ask the right questions to get your data insights projects back on track
Learn about the thinking behind endjin's Power BI Maturity assessment by applying Wardley Doctrine, and asking more questions.

My first year as an apprentice software engineer at endjin
Jessica joined endjin as part of the Software Engineering Apprenticeship 2021 cohort. In this post, Jessica reflects on her first year.
Data Engineering
View all (19)
Azure Synapse Analytics versus Microsoft Fabric: A Side by Side Comparison
In this post we compare Azure Synapse Analytics with Microsoft Fabric to understand how features map from Synapse to Fabric.

Data validation in Python: a look into Pandera and Great Expectations
Data validation is a vital step in any data-oriented workstream. This post investigates and compares two popular Python data validation packages - Pandera and Great Expectations

How to setup Python, PyEnv & Poetry on Windows
Using multiple versions of Python on Windows can be a frustrating experience, especially if you've experienced how smooth the same workflow is on MacOS and Linux/WSL. This post provides a script and guide to getting setup to use Python virtual environments and Poetry, a popular dependency management tool.
Databricks
View all (8)
Intro to Microsoft Fabric
Microsoft Fabric is a unified data & analytics platform that brings together, and improves upon, Microsoft's existing suite of data products. It can be thought of as the third generation of Microsoft data platforms, where first generation (e.g. HDInsight, SQL Data Warehouse) were somewhat isolated takes on traditional data products; second generation was Azure Synapse Analytics, which integrated platforms at a UX level but still felt a little disjointed at the data level; and now we have Microsoft Fabric which builds upon the Synapse "unification" vision, with a particular focus on enabling deep data-level interoperability. In this post we introduce its core offerings, as well as weigh up the pros and cons of this new product.

Version Control in Databricks
Databricks notebooks provide an interactive and collaborative environment for developing code while applying software engineering best practices, such as source control. This blog post provides a step by step guide of how to implement source control in Databricks.

How to use SQL Notebooks to access Azure Synapse SQL Pools & SQL on demand
Wishing Azure Synapse Analytics had support for SQL notebooks? Fear not, it's easy to take advantage rich interactive notebooks for SQL Pools and SQL on Demand.
DevOps
View all (32)
Exploring OpenChain: From License Compliance to Security Assurance
Open-source software has become an essential part of many organisation's software supply chain, however, this poses challenges with license compliance and security assurance.

Data validation in Python: a look into Pandera and Great Expectations
Data validation is a vital step in any data-oriented workstream. This post investigates and compares two popular Python data validation packages - Pandera and Great Expectations

How to setup Python, PyEnv & Poetry on Windows
Using multiple versions of Python on Windows can be a frustrating experience, especially if you've experienced how smooth the same workflow is on MacOS and Linux/WSL. This post provides a script and guide to getting setup to use Python virtual environments and Poetry, a popular dependency management tool.
Engineering Practices
View all (141)
Agile teams often get their planning and priorities wrong
Do you struggle to manage conflicting demands from multiple stakeholders, even when they have agreed about the relative priority of their requests? Apply a budgeting methodology to agile projects for better planning and communication with the team and wider stakeholders.

Rx.NET v6.0 Now Available
For the first time since 2020, a new release of Rx.NET is available, supporting .NET 6 and .NET 7.

Version Control in Databricks
Databricks notebooks provide an interactive and collaborative environment for developing code while applying software engineering best practices, such as source control. This blog post provides a step by step guide of how to implement source control in Databricks.
Innovation
View all (30)
How to Monetize APIs with Azure API Management
Monetizing your APIs is a complex challenge; creating your business, revenue, & pricing model, then translating into a technical implementation requires significant investment. Working with the Azure API Management team we provide guidance, videos and code to help accelerate your journey.

Do robots dream of counting sheep?
Some of my thoughts inspired whilst helping out on the farm over the weekend. What is the future of work given the increasing presence of machines in our day to day lives? In which situations can AI deliver greatest value? How can we ease the stress of digital transformation on people who are impacted by it?

Azure Synapse Analytics: How serverless is replacing the data warehouse
Serverless data architectures enable leaner data insights and operations. How do you reap the rewards while avoiding the potential pitfalls?
Internet of Things
View all (15)
Do robots dream of counting sheep?
Some of my thoughts inspired whilst helping out on the farm over the weekend. What is the future of work given the increasing presence of machines in our day to day lives? In which situations can AI deliver greatest value? How can we ease the stress of digital transformation on people who are impacted by it?

How to use SQL Notebooks to access Azure Synapse SQL Pools & SQL on demand
Wishing Azure Synapse Analytics had support for SQL notebooks? Fear not, it's easy to take advantage rich interactive notebooks for SQL Pools and SQL on Demand.

ArrayPool vs MemoryPool—minimizing allocations in AIS.NET
Tracking down unexpected allocations in a high-performance .NET parsing library.
Machine Learning
View all (31)
SQLbits 2023 - The Best Bits
This is a summary of the sessions I attended at SQLbits 2023 in Newport Wales, which is Europe's largest expert led data conference.

SQLbits 2022 - The Best Bits
This is a summary of the sessions I attended at SQLbits 2022 in London, which is Europe's largest expert led data conference.

Do robots dream of counting sheep?
Some of my thoughts inspired whilst helping out on the farm over the weekend. What is the future of work given the increasing presence of machines in our day to day lives? In which situations can AI deliver greatest value? How can we ease the stress of digital transformation on people who are impacted by it?
Microsoft Fabric
View all (4)
Microsoft Fabric: Announced
Microsoft Fabric extends the promise of Azure Synapse integration to all analytics workloads from the data engineer to the business knowledge worker. It brings together Power BI, Data Factory, and the Data Lake, on a new generation of the Synapse data infrastructure. Delivered as a unified SaaS offering, it aims to reduce cost and time to value, while enabling new "citizen data science" capabilities. Check out all the resources from the endjin team collated in this post.

What is OneLake?
OneLake is the core of Microsoft Fabric - it is essentially the storage account for all of your data utilized within Fabric, be that within Azure or from another cloud (through Shortcuts). As the name suggests, it is a single, logical data lake backing more or less all of your Fabric workloads. For fans of analogies: you'll have probably heard Microsoft use the tag-line "OneLake is to data what OneDrive is to files/documents". Read this blog to dive deeper into OneLake and learn about why you should be excited.

Azure Synapse Analytics versus Microsoft Fabric: A Side by Side Comparison
In this post we compare Azure Synapse Analytics with Microsoft Fabric to understand how features map from Synapse to Fabric.
Open Source
View all (61)
Rx.NET v6.0 Now Available
For the first time since 2020, a new release of Rx.NET is available, supporting .NET 6 and .NET 7.

Working with JSON in Pyspark
This blog shows how to transform a document based format to a tabular format whilst managing all the nested properties

The OpenChain specification explained
When implementing OpenChain, understanding the specification will help guide your organisation to having processes in place to review and manage open-source software
OpenChain
View all (4)
Exploring OpenChain: From License Compliance to Security Assurance
Open-source software has become an essential part of many organisation's software supply chain, however, this poses challenges with license compliance and security assurance.

The OpenChain specification explained
When implementing OpenChain, understanding the specification will help guide your organisation to having processes in place to review and manage open-source software

What are the risks with open-source software?
The key risks associated with open-source software, from whether you use it minimally, to using it throughout all your systems.
Power BI
View all (65)
Microsoft Fabric: Announced
Microsoft Fabric extends the promise of Azure Synapse integration to all analytics workloads from the data engineer to the business knowledge worker. It brings together Power BI, Data Factory, and the Data Lake, on a new generation of the Synapse data infrastructure. Delivered as a unified SaaS offering, it aims to reduce cost and time to value, while enabling new "citizen data science" capabilities. Check out all the resources from the endjin team collated in this post.

Intro to Microsoft Fabric
Microsoft Fabric is a unified data & analytics platform that brings together, and improves upon, Microsoft's existing suite of data products. It can be thought of as the third generation of Microsoft data platforms, where first generation (e.g. HDInsight, SQL Data Warehouse) were somewhat isolated takes on traditional data products; second generation was Azure Synapse Analytics, which integrated platforms at a UX level but still felt a little disjointed at the data level; and now we have Microsoft Fabric which builds upon the Synapse "unification" vision, with a particular focus on enabling deep data-level interoperability. In this post we introduce its core offerings, as well as weigh up the pros and cons of this new product.

How to develop an accessible colour palette for Power BI
If the audience for your Power BI report is larger than 12 people, then statistically 1 of your audience suffers from Colour Vision Deficiency. In this post we will talk through how we experimented with colour to develop an accessible colour palette and why colour and contrast is important when it comes to data visualisation.
Python
View all (16)
Azure Synapse Analytics versus Microsoft Fabric: A Side by Side Comparison
In this post we compare Azure Synapse Analytics with Microsoft Fabric to understand how features map from Synapse to Fabric.

Data validation in Python: a look into Pandera and Great Expectations
Data validation is a vital step in any data-oriented workstream. This post investigates and compares two popular Python data validation packages - Pandera and Great Expectations

How to setup Python, PyEnv & Poetry on Windows
Using multiple versions of Python on Windows can be a frustrating experience, especially if you've experienced how smooth the same workflow is on MacOS and Linux/WSL. This post provides a script and guide to getting setup to use Python virtual environments and Poetry, a popular dependency management tool.
Security and Compliance
View all (27)
Exploring OpenChain: From License Compliance to Security Assurance
Open-source software has become an essential part of many organisation's software supply chain, however, this poses challenges with license compliance and security assurance.

The OpenChain specification explained
When implementing OpenChain, understanding the specification will help guide your organisation to having processes in place to review and manage open-source software

What are the risks with open-source software?
The key risks associated with open-source software, from whether you use it minimally, to using it throughout all your systems.
Startups
View all (15)
How to Monetize APIs with Azure API Management
Monetizing your APIs is a complex challenge; creating your business, revenue, & pricing model, then translating into a technical implementation requires significant investment. Working with the Azure API Management team we provide guidance, videos and code to help accelerate your journey.

10 ways working with Microsoft helped endjin grow since 2010
Microsoft recently shot a video interviewing endjin co-founder, Howard van Rooijen, and Director of Engineering, James Broome, about how Microsoft has helped endjin grow over the past decade. This posts the top 10 ways in which Microsoft helped - from providing access to valuable software and services, to opening up sales channels, to helping to navigate the minefield of UK Financial Services regulations around cloud adoption.

What makes a successful FinTech start-up?
In this post we discuss the characteristics of a great FinTech startup, and the importance of the API Economy to innovation in Financial Services.
Strategy
View all (64)
Agile teams often get their planning and priorities wrong
Do you struggle to manage conflicting demands from multiple stakeholders, even when they have agreed about the relative priority of their requests? Apply a budgeting methodology to agile projects for better planning and communication with the team and wider stakeholders.

Ask the right questions to get your data insights projects back on track
Learn about the thinking behind endjin's Power BI Maturity assessment by applying Wardley Doctrine, and asking more questions.

SQLbits 2023 - The Best Bits
This is a summary of the sessions I attended at SQLbits 2023 in Newport Wales, which is Europe's largest expert led data conference.
UX
View all (19)
How to develop an accessible colour palette for Power BI
If the audience for your Power BI report is larger than 12 people, then statistically 1 of your audience suffers from Colour Vision Deficiency. In this post we will talk through how we experimented with colour to develop an accessible colour palette and why colour and contrast is important when it comes to data visualisation.

How to enable data teams with the design assets required for impactful data storytelling in Power BI
In this post we will talk through how to expand on a data team's creative skillset, without access to specialist photo editing software such as Photoshop or Illustrator.

How to dynamically switch between measures in Power BI visuals with Field Parameters
Field Parameters allow users to dynamically change the measure being analysed within a report. This blog post will explore how to use the new Field Parameters feature to dynamically switch between measures in a visua.
Visualisation
View all (12)
How to develop an accessible colour palette for Power BI
If the audience for your Power BI report is larger than 12 people, then statistically 1 of your audience suffers from Colour Vision Deficiency. In this post we will talk through how we experimented with colour to develop an accessible colour palette and why colour and contrast is important when it comes to data visualisation.

How to enable data teams with the design assets required for impactful data storytelling in Power BI
In this post we will talk through how to expand on a data team's creative skillset, without access to specialist photo editing software such as Photoshop or Illustrator.

How to dynamically switch between measures in Power BI visuals with Field Parameters
Field Parameters allow users to dynamically change the measure being analysed within a report. This blog post will explore how to use the new Field Parameters feature to dynamically switch between measures in a visua.