Analytics

Fix the following issues: Errors converting large datasets to pandas, pandas for Spark is very slow, and pandas for Spark column reduction doesn't reduce data.

Intelligently scheduling cloud data pipelines based on carbon impact can optimize both environmental sustainability and operational efficiency.

Intelligently scheduling cloud data pipelines based on carbon impact can optimize both environmental sustainability and operational efficiency.

Intelligently scheduling cloud data pipelines based on carbon impact can optimize both environmental sustainability and operational efficiency.

Intelligently scheduling cloud data pipelines based on carbon impact can optimize both environmental sustainability and operational efficiency.

Power BI's new PBIR format enhances collaboration, version control, and efficiency for developers. Learn key benefits and future implications.

Power BI Projects are a game changer for teams building reports; offering a source-control friendly format, CI/CD support, and the ability to edit in a code editor.

This guide captures the essential points that leaders should consider when setting up a new data team.

Our experience shows that the the most successful data projects rely heavily on building a multi-disciplinary team.

As data and AI become the engine of business change, we need to learn the lessons of the past to avoid expensive failures.

This is a summary of the sessions I attended at SQLbits 2024 - Europe's largest expert led data conference. This year SQLBits was hosted at Farnborough IECC, Hampshire.

Explore a step-by-step guide on designing a side nav in Power BI, covering form, icons, states, actions, with a view to enhancing report design & UI.

Learn how to access multi-select choice column choice labels from Azure Synapse Link for Dataverse using PySpark or SQL.

Learn how to access the choice labels from Azure Synapse Link for Dataverse using T-SQL through SQL Serverless and by using Spark SQL in a Synapse Notebook.

Discover a concise guide on improving Model Driven Power Apps styles with step-by-step instructions for a polished user experience.

Learn how to access the choice labels from Azure Synapse Link for Dataverse using PySpark.

This blog post explores interactive notebooks in Azure Databricks. An Azure Databricks Notebook is a powerful data science tool that support exploratory data analysis, hypothesis testing, data cleaning and transformation, data visualisation, statistical modeling and machine learning.

Explore AI-powered self-service reporting with tools like Copilot in Power BI and Microsoft Fabric, balancing benefits and pitfalls.

Microsoft Fabric unifies Power BI, Data Factory & Data Lake on Synapse infrastructure, reducing cost & time while enabling citizen data science.

Explore OneLake, Microsoft Fabric's core storage for data in Azure & other clouds. Discover its role in Fabric workloads, the OneDrive equivalent for data storage.

Microsoft Fabric unifies data & analytics, building on Azure Synapse Analytics for improved data-level interoperability. Explore its offerings & pros/cons.

Discover the use of Synapse Notebooks in Azure Synapse Analytics for data analysis, cleaning, visualization, and machine learning.

Explore how to implement source control in Databricks notebooks, promoting software engineering best practices.

Learn about the thinking behind endjin's Power BI Maturity assessment by applying Wardley Doctrine, and asking more questions.

This is a summary of the sessions I attended at SQLbits 2023 in Newport Wales, which is Europe's largest expert led data conference.

This blog shows how to transform a document based format to a tabular format whilst managing all the nested properties

Implement Python data validation with Pandera & Great Expectations in this comparison of their features and use cases.

Explore how we developed an accessible color palette for Power BI reports, considering color vision deficiency and data visualization.

Here are some of the highlights form this year's Big Data LDN conference.

Explore Custom Objects in Lake Databases for user-friendly column names, calculated columns, and pre-defined queries in Azure Synapse Analytics.

Explore Lake Databases in Azure Synapse Analytics: analyze Dataverse data, share Spark tables, and design models with Database Templates.

This blog post explores an exciting, hidden new feature in DAX - the EVALUATEANDLOG function. EVALUATEANDLOG brings debugging capability to Power BI.

Many data projects fail to deliver the impact they should for a simple reason – they focus on the data. This series of posts explains a different way of thinking that will set up your data & analytics projects for success. Using an iterative, action-oriented, insight discovery process, it demonstrates tools and techniques that will help you to identify, define and prioritize requirements in your own projects so that they deliver maximum value. It also explores the synergy with modern cloud analytics platforms like Azure Synapse, explaining how the process and the architecture actively support each other for fast, impactful delivery.

Discover a unique, action-oriented approach to data projects for maximum impact. Learn how to prioritize requirements and leverage cloud analytics platforms.

This series focuses on maximizing data projects' impact via an iterative, insight discovery process, and synergy with cloud platforms like Azure Synapse.

Discover a unique, action-oriented approach to data projects for maximum impact. Learn how to synergize with platforms like Azure Synapse for fast delivery.

In this post we will talk through how to expand on a data team's creative skillset, without access to specialist photo editing software such as Photoshop or Illustrator.

I recently passed the PL-300 - Power BI Data Analyst exam. Here are some tips to prepare for it that I found useful!

Many data projects fail to deliver the impact they should for a simple reason – they focus on the data. This series of posts explains a different way of thinking that will set up your data & analytics projects for success. Using an iterative, action-oriented, insight discovery process, it demonstrates tools and techniques that will help you to identify, define and prioritize requirements in your own projects so that they deliver maximum value. It also explores the synergy with modern cloud analytics platforms like Azure Synapse, explaining how the process and the architecture actively support each other for fast, impactful delivery.

Optimise Power BI report performance with analyzer tools. Discover essential techniques for efficient report development in this blog post.

Discover a unique, action-oriented approach to data projects for maximum impact. Learn how to synergize with platforms like Azure Synapse for fast delivery.

Automate Excel tasks with Office Scripts & Power Automate. Get an overview and explore a practical example in this post.

In this blog we demonstrate how the Gherkin specification can be adapted to enable BDD to be applied to data engineering use cases.

Learn how to grant non-admin users access to Spark synchronized objects with SQL Serverless in Synapse Analytics using the Shared Metadata Model.

Explore Azure Synapse's 'Shared Metadata Model' feature. Learn how it syncs Spark tables with SQL Serverless, its benefits, and tradeoffs.

This blog post explores how the CALCULATE function in DAX performs context transition.

Explore the impact of Excel's numeric precision rules on identifiers and learn about infamous data loss incidents caused by misuse in this post.

This blog post explores one of the most powerful functions in DAX - CALCULATE.

This blog post explores two common functions in DAX - RELATED and RELATEDTABLE.

Discover how to extract insights from spreadsheets and CSV files using Pandas and Power BI in this blog post.

Explore filtering fact tables with Star Schemas and learn a clean method for filtering dimension tables by another dimension.

In this blog post explore using Field Parameters to dynamically switch between measures in visuals, enhancing report analysis capabilities.

This blog post explores one of the most common table functions in DAX - DISTINCT.

This blog post explores two of the most common table functions in DAX - FILTER and ALL.

Explore DAX & Power BI differences between measures & calculated columns, and learn when to use each in this informative blog post.

This is a summary of the sessions I attended at SQLbits 2022 in London, which is Europe's largest expert led data conference.

Explore a step-by-step guide on designing custom buttons in Power BI, covering shape, form, icons, states, actions, and enhancing report design & UI.

Explore a simple, visual approach to prioritisation that aids decision-making and stakeholder engagement.

In this third and final part of this series, we learn how CALCULATE performs Context Transition when used inside of a filter context.

After learning about the two different types of evaluation contexts in our previous post, we now talk about table relationships and how these interact with the filter and row contexts to condition the output of our DAX code.

Explore DAX queries for scenario-based testing in Power BI reports to ensure data model validity, rule adherence, and security maintenance.

Explore translating a company's brand into Power BI reports and extending their visual identity to include the Power BI platform.

Explore DAX query language in Power BI: learn about Evaluation Contexts and their impact on code for improved formula results.

Explore the benefits of Power BI's new ExecuteQueries REST API for developers in tooling, process, and integrations.

Thinking of data as the new soil is useful in highlighting the key elements that enable a successful data and analytics initiative.

TL;DR - Using Azure Synapse SQL Serverless, you can query Azure Data Lake and populate Power BI reports across multiple workspaces.

Explore data with Azure Synapse's interactive Spark notebooks, integrated with Pipelines & monitoring tools. Learn how to add tests for business rule validation.

Some of my thoughts inspired whilst helping out on the farm over the weekend. What is the future of work given the increasing presence of machines in our day to day lives? In which situations can AI deliver greatest value? How can we ease the stress of digital transformation on people who are impacted by it?

Discover Azure Data Factory's null-safe operator for referencing activity outputs that may not always exist. Learn to use it effectively.

Modern analytics requires a multi-faceted approach, which can cause integration headaches. Azure Synapse's Swiss army knife approach can remove a lot of friction.

Data insights are useless, and even dangerous (as we've seen recently at Public Health England) if they can't be trusted. So, the need to validate business rules and security boundaries within a data solution is critical. This post argues that if you're doing anything serious with data, then you should be taking this seriously.

Fix the "You need permission" error in Azure Synapse Analytics with this guide, addressing its causes and solutions for Data Engineers/Developers.

Assign roles in Synapse Studio for Azure Synapse Analytics devs using Azure CLI. Accessible by Owners/Contributors of the resource.

Despite the subsequent media reporting, the loss of 16,000 Covid-19 test results at Public Health England wasn't caused by Excel. This post argues that a lack of an appropriate risk and mitigation analysis left the process exposed to human error, which ultimately led to the loss of data and inaccurate reporting. It describes a simple process that could have been applied to prevent the error, and how it will help if you're worried about ensuring quality or reducing risk in your own business, technology or data programmes.

Explore Azure Synapse Link for Cosmos DB's impact on 'full stack serverless', No-ETL, and pay-as-you-query analytics.

Wishing Azure Synapse Analytics had support for SQL notebooks? Fear not, it's easy to take advantage rich interactive notebooks for SQL Pools and SQL on Demand.

Explore deploying Azure Synapse Analytics workspaces using ARM templates, a popular infrastructure deployment method for organizations.

Serverless data architectures enable leaner data insights and operations. How do you reap the rewards while avoiding the potential pitfalls?

Here is a quick tip on how to remove column summarisation from a matrix in Power BI!

Here's a quick tip on how to alter the sort order of a column in Power BI!

Boost Power BI report readability with dynamic font colors for diverse backgrounds, ensuring clear text display and enhanced accessibility.

New Azure Synapse Analytics service offers SQL Serverless for on-demand data lake queries. We tested its potential as a Data Lake Analytics replacement.

Did you know that Azure Synapse has great support for .NET and #csharp? Learning new languages is often a barrier to digital transformation, being able to use existing people, skills, tools and engineering disciplines can be a massive advantage.

Explore 5 key reasons to choose Azure Synapse Analytics for your cloud data needs, based on years of experience in driving customer outcomes.

Explore Power BI Embedded for ISVs, using JavaScript library for personalization, Row-level Security, and modifying Embed Requests for data filtering.

Explore how to configure model properties in Power BI for enhanced discoverability and improved data visualisation support.

Whilst "read/write XMLA endpoint" might seem like a technical mouthful, its addition to Power BI is a significant milestone in the strategy of bringing Power BI and Analysis Services closer together. As well as closing the gap between IT-managed workloads and self-service BI, it presents a number of new opportunities for Power BI developers in terms of tooling, process and integrations. This post highlights some of the key advantages of this new capability and what they mean for the Power BI developer.

This is the final blog in a series about DAX and Power BI. This post focuses on the CALCULATE function, which is a unique function in DAX. The CALCULATE function has the ability to alter filter contexts, and therefore can be used to enable extremely powerful and complex processing. This post covers some of the most common scenarios for using CALCULATE, and some of the gotchas in the way in which these different features interact!

Ensure Power BI report quality by connecting to tabular models, executing scenario-based specs, and validating data, business rules, and security.

Explore data modelling in Power BI, including loading, shaping, and enhancing data. Learn key steps and practical examples.

This is the sixth blog in a series about DAX and Power BI. This post focuses on relationships and related tables. These relationships allow us to build up intricate and powerful models using a combination of sources and tables. The use of relationships in DAX powers many of the features around slicing and page filtering of reports.

Jess and Carmel recently gave a talk at Azure Oxford on Combatting illegal fishing with Machine Learning and Azure - for less than £10 / month. The recording of that talk is now available for viewing!The talk focuses on the recent work we completed with OceanMind. They run through how to construct a cloud-first architecture based on serverless and data analytics technologies and explore the important principles and challenges in designing this kind of solution. Finally, we see how the architecture we designed through this process not only provides all the benefits of the cloud (reliability, scalability, security), but because of the pay-as-you-go compute model, has a compute cost that we could barely believe!

Ensure reliable insights with endjin's automated quality gates for validating Power BI Dataflows in complex solutions.

This is the fifth blog in a series on DAX and Power BI. This post focuses on table functions. In DAX, table functions return a table which can then be used for future processing. This can be useful if, for example, you want to perform an operation over a filtered dataset. Table functions, like most functions in DAX, operate under the filter context in which they are applied.

Explore Azure Analysis Services for scalable analytics. Control costs via automation with Powershell & Azure DevOps in multi-environment setups.

This is the fourth blog in a series about DAX and Power BI. We have so far covered filter and row contexts, and the difference between calculated columns and measures. This post focuses on aggregators. We cover the limitations of the classic aggregators, and demonstrate the power of the iterative versions. We also highlight some of the less intuitive features around how these functions interact with both filter and row contexts.

Discover how to programmatically retrieve Power BI Dataflow refresh history for automated operations, enabling efficient dataset refresh management.

Learn how to update Azure Analysis Services model schemas in .NET apps using AMO SDK for user-driven analysis.

Wardley Maps are a fantastic tool to help provide situational awareness, in order to help you make better decisions. We use Wardley Maps to help our customers think about the various benefits and trade-offs that can be made when migrating to the Cloud. In this blog post, Jess Panni demonstrates how we used Wardley Maps to plan the migration of OceanMind to Microsoft Azure, and how the maps highlighted where the core value of their platform was, and how PaaS and Serverless services offered the most value for money for the organisation.

Dive into DAX and Power BI in our blog. Learn about calculated columns, measures, and their use in complex visuals.

Tour our OceanMind project using Azure Functions for real-time marine telemetry processing. Learn optimization techniques for serverless environments.

Here is the second blog in a series around learning DAX and Power BI. This post focuses on row contexts, which are used when iterating over the rows of a table when, for example, evaluating a calculated column. Row contexts along with filter contexts underpin the basis of the DAX language. Once you understand this underlying theory it is purely a case of learning the syntax for the different operations which are built on top of it.

Learn to use Azure Analysis Services in custom apps via REST API in .NET for efficient async model refreshes.

Explore the basics of DAX and Power BI in this series. This first post focuses on filter contexts, a key concept for effective DAX usage.

Explore endless possibilities with dynamic DAX queries in C# for Azure Analysis Services integration in custom apps using the provided code samples.

The theme of this year's British Science Week (6 - 15 March 2020) is "Our Diverse Planet". We'll be getting involved by speaking to school children about the work we've been doing with Oxfordshire-based OceanMind (part of the Microsoft AI for Good programme) to help them combat illegal fishing, hopefully inspiring some of the next generation of data scientists!

Explore .NET querying methods for integrating Azure Analysis Services beyond data querying into dynamic UIs and APIs.

Learn to integrate Azure Analysis Services in apps by establishing server connections. Follow this guide with code samples for essential scenarios.

Ed attended NDC London 2020, along with many of his endjin colleagues. In this post he summarises and reflects upon his favourite sessions of the conference including; "OWASP Top Ten proactive controls" by Jim Manico, "There's an Impostor in this room!" by Angharad Edwards, "How to code music?" by Laura Silvanavičiūtė, "ML and the IoT: Living on the Edge" by Brandon Satrom, "Common API Security Pitfalls" by Philippe De Ryck, and "Combatting illegal fishing with Machine Learning and Azure - for less than £10 / month" by Jess Panni & Carmel Eve.

Explore Azure Analysis Services in custom apps using SDKs, PowerShell cmdlets & REST APIs. Learn to choose the right framework in this guide.

Explore Azure Analysis Services' versatility in bespoke analysis products & processes, and learn how it can unlock data insights beyond traditional BI.

In this blog we discuss building a secure data solution using Azure Data Lake. Data Lake has many features which enable fine grained security and data separation. It is also built on Azure Storage which enables us to take advantage of all of those features and means that ADLS is still a cost effective storage option!This post runs through some of the great features of ADLS and runs through an example of how we build our solutions using this technology!

In January 2020, Carmel is speaking about creating high performance geospatial algorithms in C# which can detect suspicious vessel activity, which is used to help alert law enforcement to illegal fishing. The input data is fed from Azure Data Lake Storage Gen 2, and converted into data projections optimised for high-performance computation. This code is then hosted in Azure Functions for cheap, consumption based processing.

Learn to import/export notebooks in Databricks workspaces manually or programmatically, and transfer content between workspaces efficiently.

Machine learning often seems like a black box. This post walks through what's actually happening under the covers, in an attempt to de-mystify the process!Neural networks are built up of neurons. In a shallow neural network we have an input layer, a "hidden" layer of neurons, and an output layer. For deep learning, there is simply more hidden layers which allows for combining neuron's inputs and outputs to build up a more detailed picture.If you have an interest in Machine Learning and what is really happening, definitely give this a read (WARNING: Some algebra ahead...)!

Automate Power BI workspaces creation & security using PowerShell & Azure DevOps pipelines in a wider DevOps strategy with Azure data platform services.

Explore data analysis & ETL with Databricks Notebooks on Azure. Utilize Spark's unified analytics engine for big data & ML, and integrate with ADF pipelines.

Snowflake is a cloud native data warehouse platform, that enabled data engineering, data science, data lakes, data sharing and data warehousing. Endjin are very excited to announce our partnership.

Mapping Data Flows are a relatively new feature of ADF. They allow you to visually build up complex data transformation sequences. This can aid in the streamlining of data manipulation and ETL processes, without the need to write any code! This post gives a brief introduction to the technology, and what this could enable!

Explore the lack of a native Azure Data Factory connector for Snowflake and discover alternatives for integration between these popular platforms.

Power BI has become more successful than anyone anticipated. We've decided to spin out the Power BI category from the Azure Weekly Newsletter into it's own publication - Power BI Weekly.

When he joined endjin, Technical Fellow Ian sat down with founder Howard for a Q&A session. This was originally published on LinkedIn in 5 parts, but is republished here, in full. Ian talks about his path into computing, some highlights of his career, the evolution of the .NET ecosystem, AI, and the software engineering life.

Ian Griffiths recently joined endjin as a Technical Fellow. We had a long fireside chat, which has been broken down into a 5 part series covering .NET, the Cloud, AI, ML, teaching software engineering, and why he joined endjin.

As an apprentice engineer at endjin, you cover a lot of ground, especially at a consultancy which works with the latest and greatest tools to solve its clients' problems. Learn about endjin's Modern Data Platform, which is a culmination of IP, processes and knowledge built from years of implementing high-performance data-driven solutions. Also learn about the types of tools an apprentice gets to use, and the types of things an apprentice learns along the way.

Performing Power BI date table generation with CALENDAR & CALENDARAUTO functions. Learn key considerations for time-intelligence calculations in reports.

Learn to use SQL Server's Python integration for efficient data handling. Eliminate clunky transfers and easily operationalize Python models/scripts.

Learn what types of things an apprentice gets up to at endjin a few months after joining. You could be learning about Neural Networks: algorithms which mimic the way biological systems process information. You could be attending Microsoft's Future Decoded conference, learning about Bots, CosmosDB, IoT and much more. Hopefully, you wouldn't be in hospital after a ruptured appendix!

Read how we set up Azure spending alerts and saved over $10k

Structured apprenticeships provide a great way to build skills whilst getting real-life experience. Endjin's apprenticeship scheme has been refined over years, with an optimal mixture of training, project work, and exposure to commercial processes - a scheme which is designed to build strong foundations for a well-rounded Software Engineering consultant. This post explains the transition from university to an apprenticeship at endjin, including the types of work an apprentice could end up doing, and some examples of real-life learnings from a real-life apprentice.

Learn to build a Power BI dashboard using DirectQuery and ElasticQuery across multiple databases with Alice Waddicor.

A career in software engineering doesn't need to start with a Computer Science degree. The underlying traits of problem solving, a willingness to learn and the ability to collaborate well can be built in any field. Internships provide a great way to get your foot-in-the-door in the professional world, and arm you with some real-life experience for future endeavours. This post describes an internship at endjin, including the type of work you could be asked to do and what you could learn.

Discover Azure DevOps Work Items features, use RESTful API for insights, and Power BI visualization in our step-by-step guide.

Many organisations are starting to adopt the R Programming Language for their data science and financial modelling scenarios. But just because the language is being used for modelling, doesn't mean you should write unit tests that can be exercised as part of your CI/CD pipeline. In this blog post Jess Panni demonstrates how you can run R unit tests inside Azure DevOps.

Meta-Runners allow you to easily create reusable build components for TeamCity, in this post I demonstrate how to create a Meta-Runner to create Azure Application Insights Release Annotations.

What do machine learning and data science actually mean? This post digs into the detail behind the endjin approach to structured experimentation, arguing that the "science" is really all about following the process, allowing you to iterate to insights quickly when there are no guarantees of success.

We have produced an insightful booklet called "Embracing Disruption - Financial Services and the Microsoft Cloud" which examines the challenges and opportunities for the Financial Service Industry in the UK, through the lens of Microsoft Azure, Security, Privacy & Data Sovereignty, Data Ingestion, Transformation & Enrichment, Big Compute, Big Data, Insights & Visualisation, Infrastructure, Ops & Support, and the API Economy.

This post looks at what machine learning really is (and isn't), dispelling some of the myths and hype that have emerged as the interest in data science, predictive analytics and machine learning has grown. Without any hard guarantees of success, it argues that machine learning as a discipline is simply trial and error at scale – proving or disproving statistical scenarios through structured experimentation.