Skip to content
James Broome - Director of Engineering

James Broome

Director of Engineering

James knows a thing or two about successful project delivery. He is consistently seen as a 'safe pair of hands' by customers, as he ensures our people and processes are always running smoothly. He thinks Director of Engineering is a rather grandiose job title, but is coming to terms with that.

Meet James

James Broome

James has spent nearly 20 years delivering high quality software solutions addressing global business problems, with teams and clients across 3 continents. As Director of Engineering at endjin, he leads the team in providing technology strategy, data insights and engineering support to organisations of all sizes - from disruptive B2C start-ups, to global financial institutions. He's responsible for the success of our customer-facing project delivery, as well as the capability and growth of our delivery team.

James’ blog

Insight Discovery (part 6) – How to define business requirements for a successful cloud data & analytics project

Insight Discovery (part 6) – How to define business requirements for a successful cloud data & analytics project

James Broome

Many data projects fail to deliver the impact they should for a simple reason – they focus on the data. This series of posts explains a different way of thinking that will set up your data & analytics projects for success. Using an iterative, action-oriented, insight discovery process, it demonstrates tools and techniques that will help you to identify, define and prioritize requirements in your own projects so that they deliver maximum value. It also explores the synergy with modern cloud analytics platforms like Azure Synapse, explaining how the process and the architecture actively support each other for fast, impactful delivery.
Insight Discovery (part 5) – Deliver insights incrementally with data pipelines

Insight Discovery (part 5) – Deliver insights incrementally with data pipelines

James Broome

Many data projects fail to deliver the impact they should for a simple reason – they focus on the data. This series of posts explains a different way of thinking that will set up your data & analytics projects for success. Using an iterative, action-oriented, insight discovery process, it demonstrates tools and techniques that will help you to identify, define and prioritize requirements in your own projects so that they deliver maximum value. It also explores the synergy with modern cloud analytics platforms like Azure Synapse, explaining how the process and the architecture actively support each other for fast, impactful delivery.
Insight Discovery (part 4) – Data projects should have a backlog

Insight Discovery (part 4) – Data projects should have a backlog

James Broome

Many data projects fail to deliver the impact they should for a simple reason – they focus on the data. This series of posts explains a different way of thinking that will set up your data & analytics projects for success. Using an iterative, action-oriented, insight discovery process, it demonstrates tools and techniques that will help you to identify, define and prioritize requirements in your own projects so that they deliver maximum value. It also explores the synergy with modern cloud analytics platforms like Azure Synapse, explaining how the process and the architecture actively support each other for fast, impactful delivery.
Insight Discovery (part 3) – Defining Actionable Insights

Insight Discovery (part 3) – Defining Actionable Insights

James Broome

Many data projects fail to deliver the impact they should for a simple reason – they focus on the data. This series of posts explains a different way of thinking that will set up your data & analytics projects for success. Using an iterative, action-oriented, insight discovery process, it demonstrates tools and techniques that will help you to identify, define and prioritize requirements in your own projects so that they deliver maximum value. It also explores the synergy with modern cloud analytics platforms like Azure Synapse, explaining how the process and the architecture actively support each other for fast, impactful delivery.
Insight Discovery (part 2) – successful data projects start by forgetting about the data

Insight Discovery (part 2) – successful data projects start by forgetting about the data

James Broome

Many data projects fail to deliver the impact they should for a simple reason – they focus on the data. This series of posts explains a different way of thinking that will set up your data & analytics projects for success. Using an iterative, action-oriented, insight discovery process, it demonstrates tools and techniques that will help you to identify, define and prioritize requirements in your own projects so that they deliver maximum value. It also explores the synergy with modern cloud analytics platforms like Azure Synapse, explaining how the process and the architecture actively support each other for fast, impactful delivery.
Insight Discovery (part 1) – why do data projects often fail?

Insight Discovery (part 1) – why do data projects often fail?

James Broome

Many data projects fail to deliver the impact they should for a simple reason – they focus on the data. This series of posts explains a different way of thinking that will set up your data & analytics projects for success. Using an iterative, action-oriented, insight discovery process, it demonstrates tools and techniques that will help you to identify, define and prioritize requirements in your own projects so that they deliver maximum value. It also explores the synergy with modern cloud analytics platforms like Azure Synapse, explaining how the process and the architecture actively support each other for fast, impactful delivery.
Testing Power BI Reports with the ExecuteQueries REST API

Testing Power BI Reports with the ExecuteQueries REST API

James Broome

Despite being inherently difficult to test, the need to validate data modelling, business rules and security boundaries in Power BI reports is important, as well as the need for ensuring that quality doesn't regress over time as the insights evolve. This post explains that, by executing DAX queries over underlying tabular model, it is possible to execute scenario-based specifications to add quality gates and build confidence in Power BI reports, just as any other software project.
Why you should care about the new Power BI ExecuteQueries API

Why you should care about the new Power BI ExecuteQueries API

James Broome

The new Power BI ExecuteQueries REST API presents a number of new opportunities for Power BI developers in terms of tooling, process and integrations. This post highlights some of the key advantages of this new capability.
Managing schemas in Azure Synapse SQL Serverless

Managing schemas in Azure Synapse SQL Serverless

James Broome

SQL Serverless in Azure Synapse provides a structured way to query your data on-demand directly from your data lake. This consumption-based, flexible approach to data warehousing provides a compelling alternative to the traditional star-schema or RDBMS, but comes with it's own set of new challenges. This post explores some of the considerations around managing schemas in a serverless world, including an approach for ensuring that your data and schema remain in sync.
How to test Azure Synapse notebooks

How to test Azure Synapse notebooks

James Broome

Interactive Spark notebooks are an incredibly powerful tool for data exploration and experimentation. And in Azure Synapse, the time to (business) value is significantly decreased due to tight integration with Pipelines and monitoring tooling. But as with any software process, the need to validate business rules is important, as is ensuring that quality doesn't regress over time - especially so in such a collaborative and productive environment. This post looks at some simple ways to add testing to your Synapse Notebooks.
How do I know if my data solutions are accurate?

How do I know if my data solutions are accurate?

James Broome

Data insights are useless, and even dangerous (as we've seen recently at Public Health England) if they can't be trusted. So, the need to validate business rules and security boundaries within a data solution is critical. This post argues that if you're doing anything serious with data, then you should be taking this seriously.
The Public Health England Test and Trace Excel error could have been prevented by this one simple step

The Public Health England Test and Trace Excel error could have been prevented by this one simple step

James Broome

Despite the subsequent media reporting, the loss of 16,000 Covid-19 test results at Public Health England wasn't caused by Excel. This post argues that a lack of an appropriate risk and mitigation analysis left the process exposed to human error, which ultimately led to the loss of data and inaccurate reporting. It describes a simple process that could have been applied to prevent the error, and how it will help if you're worried about ensuring quality or reducing risk in your own business, technology or data programmes.
Does Azure Synapse Link redefine the meaning of full stack serverless?

Does Azure Synapse Link redefine the meaning of full stack serverless?

James Broome

Azure Synapse Link for Cosmos DB is a game-changing piece in the Synapse suite of services - extending the support for SQL on Demand to enable querying over the Cosmos DB Analytical Store. This post explores whether the term 'full stack serverless' should now be extended to cover No-ETL and pay-as-you-query analytics, alongside serverless application architectures.
How to use Axios interceptors to poll for long running API calls

How to use Axios interceptors to poll for long running API calls

James Broome

Many API frameworks implement an async HTTP API pattern to address the problem of coordinating the state of long-running operations with external clients. This post looks at how we can use Axios interceptors to simplify and centralise the UI logic required to work with this API pattern.
A simple invite user flow for AAD B2C (without custom policies)

A simple invite user flow for AAD B2C (without custom policies)

James Broome

Azure AD B2C is easy to integrate and configure for simple sign-in / sign-up authentication. But there's a step change in complexity as soon as you want to do something outside of the built-in user flows. An "invite user" flow is one such example, but it's also a fairly common requirement in any business or team orientated SaaS application, which makes AAD B2C as less attractive choice. This post describes an alternative approach for this authentication feature that avoids the need for AAD B2C custom policies, allowing you to keep authentication as simple as possible.
Azure Synapse Analytics: How serverless is replacing the data warehouse

Azure Synapse Analytics: How serverless is replacing the data warehouse

James Broome

Serverless data architectures enable leaner data insights and operations. How do you reap the rewards while avoiding the potential pitfalls?
Benchmarking Azure Synapse Analytics - SQL Serverless, using Polyglot Notebooks

Benchmarking Azure Synapse Analytics - SQL Serverless, using Polyglot Notebooks

James Broome

There is a new service in town that promises to transform the way you query the contents of your data lake. Azure Synapse Analytics comes with a new offering called SQL Serverless allowing you to query your data on-demand with no need for pre-provisioned resources.When we heard about the new service we were keen to get involved, so for the last 10 months we've been working with the SQL Serverless product group to provide feedback on the service and to help ensure it meets our customers needs. During this time we've put it through it's paces by implementing a range of real-world use cases. We were particularly interested to see how it stacked up as a replacement for Data Lake Analytics, where to date there has been no clear and easy migration path.
Azure Synapse for C# Developers: 5 things you need to know

Azure Synapse for C# Developers: 5 things you need to know

James Broome

Did you know that Azure Synapse has great support for .NET and #csharp? Learning new languages is often a barrier to digital transformation, being able to use existing people, skills, tools and engineering disciplines can be a massive advantage.
Does Azure Synapse Analytics spell the end for Azure Databricks?

Does Azure Synapse Analytics spell the end for Azure Databricks?

James Broome

Have you or are you about to invest in Azure Databricks? If so, the new Spark offering in Azure Synapse Analytics is likely to have grabbed your attention and rightly so. Why is Microsoft putting yet another Spark offering on the table and what does it mean for you?
5 Reasons why Azure Synapse Analytics should be on your roadmap

5 Reasons why Azure Synapse Analytics should be on your roadmap

James Broome

For years we have been building modern cloud data solutions on Azure and helping our customers transform their use of data to drive outcomes. Here are 5 reasons why Azure Synapse Analytics might just be the service that we have been crying out for.
Why Power BI developers should care about the new read/write XMLA endpoint

Why Power BI developers should care about the new read/write XMLA endpoint

James Broome

Whilst "read/write XMLA endpoint" might seem like a technical mouthful, its addition to Power BI is a significant milestone in the strategy of bringing Power BI and Analysis Services closer together. As well as closing the gap between IT-managed workloads and self-service BI, it presents a number of new opportunities for Power BI developers in terms of tooling, process and integrations. This post highlights some of the key advantages of this new capability and what they mean for the Power BI developer.
Testing Power BI Reports using SpecFlow and .NET

Testing Power BI Reports using SpecFlow and .NET

James Broome

Despite being inherently difficult to test, the need to validate data modelling, business rules and security boundaries in Power BI reports is important, as well as the need for ensuring that quality doesn't regress over time as the insights evolve. This post explains that, by connecting to the underlying tabular model, it is possible to execute scenario-based specifications to add quality gates and build confidence in Power BI reports, just as any other software project.
Testing Power BI Dataflows using SpecFlow and the Common Data Model

Testing Power BI Dataflows using SpecFlow and the Common Data Model

James Broome

Whilst testing Power BI Dataflows isn't something that many people think about, it's critical that business rules and associated data preparation steps are validated to ensure the right insights are available to the right people across the organisation. Data insights are useless, even dangerous, if they can't be trusted, so despite the lack of "official support" or recommended approaches from Microsoft, endjin treat Power BI solutions just as any other software project with respect to testing - building automated quality gates into the end to end development process. This post outlines an approach that endjin has used to test Power BI Dataflows to add quality gates and build confidence in large and complex Power BI solutions.
Azure Analysis Services - how to save money with automatic shutdown

Azure Analysis Services - how to save money with automatic shutdown

James Broome

Azure Analysis Services provides an enterprise-grade analytical platform with massive scale and flexibility. But, as one of the more expensive services in the Azure platform, consideration should be given to cost management, especially in multi-environment ALM scenarios. This post explains how to massively reduce running costs through automation using Powershell and orchestration tools like Azure DevOps.
Azure Analysis Services: How to update the expression for a calculated column from .NET

Azure Analysis Services: How to update the expression for a calculated column from .NET

James Broome

This post explains how to update Azure Analysis Services model schemas from inside custom .NET applications. Whilst not a common scenario for most, it shows that this is easy to do using the AMO SDK. So, there's nothing stopping you from developing complex and rich end-user functionality over the top of your data analysis solutions - providing run-time, user-driven schema changes like "what if" analysis.
Azure Analysis Services - How to process an asynchronous model refresh from .NET

Azure Analysis Services - How to process an asynchronous model refresh from .NET

James Broome

Integrating Azure Analysis Services into custom applications doesn't just mean read-only data querying. But if your application changes the underlying model, it will need to be re-processed before the changes take effect. This post describes how to use the REST API for Azure Analysis Services inside a custom .NET application to perform asynchronous model refreshes, meaning your applications can reliably and efficiently deal with model updates.
Azure Analysis Services: How to execute a DAX query from .NET

Azure Analysis Services: How to execute a DAX query from .NET

James Broome

Being able to construct DAX queries dynamically in C# means the possibilities are endless in terms of integrating Azure Analysis Services queries into your custom applications, and with the code samples in this post, you have everything you need to get started.
British Science Week - inspiring the next generation of data scientists

British Science Week - inspiring the next generation of data scientists

James Broome

The theme of this year's British Science Week (6 - 15 March 2020) is "Our Diverse Planet". We'll be getting involved by speaking to school children about the work we've been doing with Oxfordshire-based OceanMind (part of the Microsoft AI for Good programme) to help them combat illegal fishing, hopefully inspiring some of the next generation of data scientists!
Azure Analysis Services - How to query all the measures in a model from .NET

Azure Analysis Services - How to query all the measures in a model from .NET

James Broome

Integrating Azure Analysis Services into custom applications means more than just querying the data. By surfacing the metadata in your models, you can build dynamic and customisable UIs and APIs, tailored to the needs of the client application. This post explains how easy it is to query model metadata from .NET, so you can create deeper integrations between your data insights and your custom applications.
Azure Analysis Services: How to open a connection from .NET

Azure Analysis Services: How to open a connection from .NET

James Broome

One of the first steps in integrating Azure Analysis Services into your applications is creating and opening a connection to the server - just like any other database technology. This post explains the ins and outs of creating Azure Analysis Services connections, including code samples for each of the key scenarios.
Azure Analysis Services - integration options using .NET, REST APIs and PowerShell

Azure Analysis Services - integration options using .NET, REST APIs and PowerShell

James Broome

With a variety of integration support through client SDKs, PowerShell cmdlets and REST APIs, it can be hard to know where to start with integrating Azure Analysis Services into your custom applications. This posts walks through the options, and lays out a simple guide to choosing the right framework.
Azure Analysis Services: 8 reasons why you might want to integrate into a custom application

Azure Analysis Services: 8 reasons why you might want to integrate into a custom application

James Broome

We've done a lot of work at endjin with Azure Analysis Services over the last couple of years - but none of it has been what you'd call "traditional BI". We've pulled, twisted and bent it in all sorts of directions, using it's raw analytical processing power to underpin bespoke analysis products and processes. This post explains some of the common (and not-so-common) reasons why you might want to do similar things, and how Azure Analysis Services might be the key to unlocking your data insights.
How to use the AzureAD module in PowerShell Core

How to use the AzureAD module in PowerShell Core

James Broome

Whilst some of the Azure Active Directory PowerShell for Graph module (AzureAD) functionality has been rolled into the new Azure PowerShell Az module, it's not currently (and might never be) a replacement for the full power of what you can achieve with AzureAD. So, there's every chance you'll find yourself needing to use both side-by-side. This post explains how to do that using the new cross-platform PowerShell Core.
How to create a Power BI workspace in an Azure DevOps Pipeline using Powershell

How to create a Power BI workspace in an Azure DevOps Pipeline using Powershell

James Broome

A Power BI based solution typically consists of a variety of technologies - for example Azure data platform services containing source data. As such, automation of Power BI resources needs to be considered as part of a wider DevOps strategy. This post describes the specific steps needed in order to fully automate the creation and security of Power BI workspaces using Powershell and Azure DevOps pipelines.
Querying the Azure DevOps Work Items API directly from Power BI

Querying the Azure DevOps Work Items API directly from Power BI

James Broome

Azure DevOps Work Items offer a lot of power and features out of the box, but sometimes you need insights that Azure DevOps doesn't natively provide. In this blog post Director of Engineering, James Broome, shows how you can use the Azure DevOps Restful API to generate insights and even use Power BI to visualise them in this step-by-step guide.
"But it works on my cloud!" - are your developers still making the same mistakes in a world of DevOps and PaaS services?

"But it works on my cloud!" - are your developers still making the same mistakes in a world of DevOps and PaaS services?

James Broome

In the world of DevOps, cloud and platform services, how does a developer's "definition of done" need to change? This post argues that as the silos of development and operations are broken down, the responsibility of understanding the whole solution increases meaning, to truly take advantage of the cloud, the need for quality and professionalism is critical for success.
Machine Learning - the process is the science

Machine Learning - the process is the science

James Broome

What do machine learning and data science actually mean? This post digs into the detail behind the endjin approach to structured experimentation, arguing that the "science" is really all about following the process, allowing you to iterate to insights quickly when there are no guarantees of success.
Machine Learning - mad science or a pragmatic process?

Machine Learning - mad science or a pragmatic process?

James Broome

This post looks at what machine learning really is (and isn't), dispelling some of the myths and hype that have emerged as the interest in data science, predictive analytics and machine learning has grown. Without any hard guarantees of success, it argues that machine learning as a discipline is simply trial and error at scale – proving or disproving statistical scenarios through structured experimentation.
Azure Batch - Time is Money in Big Compute

Azure Batch - Time is Money in Big Compute

James Broome

Consumption based pricing is a one of the USPs of Cloud PaaS services, but the default settings aren't necessarily optimised for cost. Significant savings can be made from understanding your workload.