Big Data

Intelligently scheduling cloud data pipelines based on carbon impact can optimize both environmental sustainability and operational efficiency.

Intelligently scheduling cloud data pipelines based on carbon impact can optimize both environmental sustainability and operational efficiency.

Intelligently scheduling cloud data pipelines based on carbon impact can optimize both environmental sustainability and operational efficiency.

Intelligently scheduling cloud data pipelines based on carbon impact can optimize both environmental sustainability and operational efficiency.

This guide captures the essential points that leaders should consider when setting up a new data team.

Our experience shows that the the most successful data projects rely heavily on building a multi-disciplinary team.

As data and AI become the engine of business change, we need to learn the lessons of the past to avoid expensive failures.

This is a summary of the sessions I attended at SQLbits 2024 - Europe's largest expert led data conference. This year SQLBits was hosted at Farnborough IECC, Hampshire.

The first step on the road to implementing observability in your Python notebooks is basic logging. In this post, we look at how you can use Python's built in logging inside a Synapse notebook.

In this post I describe how adopting a product mindset will help you to extract maximum value from Microsoft Fabric.

Explore building situational awareness and leveraging strategic opportunities with Microsoft Fabric in this concise overview.

Explore Microsoft Fabric, inspired by Data Mesh, for a data-driven strategy. Learn to approach a Data Mesh vision using this powerful tool.

Explore Data Mesh's influence on Microsoft Fabric, addressing gaps in data product marketplace, standards, master data management, and governance.

Creating a successful organisation-wide data and analytics platform isn't just about architecture, schemas and semantic models. It's also about culture, organisational design and people. This blog explores the socio-technical nature of data and analytics and how this should influence your approach to adoption of Microsoft Fabric.

Explore AI-powered self-service reporting with tools like Copilot in Power BI and Microsoft Fabric, balancing benefits and pitfalls.

Microsoft Fabric unifies Power BI, Data Factory & Data Lake on Synapse infrastructure, reducing cost & time while enabling citizen data science.

Explore OneLake, Microsoft Fabric's core storage for data in Azure & other clouds. Discover its role in Fabric workloads, the OneDrive equivalent for data storage.

In this Microsoft Fabric vs Synapse comparison we examine how features map from Azure Synapse to Fabric.

Microsoft Fabric unifies data & analytics, building on Azure Synapse Analytics for improved data-level interoperability. Explore its offerings & pros/cons.

Learn about the thinking behind endjin's Power BI Maturity assessment by applying Wardley Doctrine, and asking more questions.

This is a summary of the sessions I attended at SQLbits 2023 in Newport Wales, which is Europe's largest expert led data conference.

Implement Python data validation with Pandera & Great Expectations in this comparison of their features and use cases.

Explore Custom Objects in Lake Databases for user-friendly column names, calculated columns, and pre-defined queries in Azure Synapse Analytics.

Explore Azure Synapse Analytics Database Templates and learn to create semantic models in this 2nd blog of the series.

Explore Lake Databases in Azure Synapse Analytics: analyze Dataverse data, share Spark tables, and design models with Database Templates.

Many data projects fail to deliver the impact they should for a simple reason – they focus on the data. This series of posts explains a different way of thinking that will set up your data & analytics projects for success. Using an iterative, action-oriented, insight discovery process, it demonstrates tools and techniques that will help you to identify, define and prioritize requirements in your own projects so that they deliver maximum value. It also explores the synergy with modern cloud analytics platforms like Azure Synapse, explaining how the process and the architecture actively support each other for fast, impactful delivery.

Explore Azure Synapse Analytics Database Templates and learn to leverage them in modern data pipelines.

Discover a unique, action-oriented approach to data projects for maximum impact. Learn how to prioritize requirements and leverage cloud analytics platforms.

This series focuses on maximizing data projects' impact via an iterative, insight discovery process, and synergy with cloud platforms like Azure Synapse.

Discover a unique, action-oriented approach to data projects for maximum impact. Learn how to synergize with platforms like Azure Synapse for fast delivery.

Many data projects fail to deliver the impact they should for a simple reason – they focus on the data. This series of posts explains a different way of thinking that will set up your data & analytics projects for success. Using an iterative, action-oriented, insight discovery process, it demonstrates tools and techniques that will help you to identify, define and prioritize requirements in your own projects so that they deliver maximum value. It also explores the synergy with modern cloud analytics platforms like Azure Synapse, explaining how the process and the architecture actively support each other for fast, impactful delivery.

Discover a unique, action-oriented approach to data projects for maximum impact. Learn how to synergize with platforms like Azure Synapse for fast delivery.

In this blog we demonstrate how the Gherkin specification can be adapted to enable BDD to be applied to data engineering use cases.

Learn how to grant non-admin users access to Spark synchronized objects with SQL Serverless in Synapse Analytics using the Shared Metadata Model.

Explore Azure Synapse's 'Shared Metadata Model' feature. Learn how it syncs Spark tables with SQL Serverless, its benefits, and tradeoffs.

Explore the impact of Excel's numeric precision rules on identifiers and learn about infamous data loss incidents caused by misuse in this post.

This is a summary of the sessions I attended at SQLbits 2022 in London, which is Europe's largest expert led data conference.

Explore a simple, visual approach to prioritisation that aids decision-making and stakeholder engagement.

Explore DAX queries for scenario-based testing in Power BI reports to ensure data model validity, rule adherence, and security maintenance.

Explore the benefits of Power BI's new ExecuteQueries REST API for developers in tooling, process, and integrations.

Explore Azure Synapse's SQL Serverless for on-demand data lake queries, its benefits, and challenges in managing schemas and maintaining data sync.

Thinking of data as the new soil is useful in highlighting the key elements that enable a successful data and analytics initiative.

Explore data with Azure Synapse's interactive Spark notebooks, integrated with Pipelines & monitoring tools. Learn how to add tests for business rule validation.

Some of my thoughts inspired whilst helping out on the farm over the weekend. What is the future of work given the increasing presence of machines in our day to day lives? In which situations can AI deliver greatest value? How can we ease the stress of digital transformation on people who are impacted by it?

Discover Azure Data Factory's null-safe operator for referencing activity outputs that may not always exist. Learn to use it effectively.

Summary of key themes from the Doing Data Together conference hosted virtually by The Scotsman newspaper and Edinburgh University in November 2020. The conference agenda was pivoted to focus on the use of data to help tackle the Covid-19 pandemic. It provided a fascinating insight into the lessons learned.

Modern analytics requires a multi-faceted approach, which can cause integration headaches. Azure Synapse's Swiss army knife approach can remove a lot of friction.

Data insights are useless, and even dangerous (as we've seen recently at Public Health England) if they can't be trusted. So, the need to validate business rules and security boundaries within a data solution is critical. This post argues that if you're doing anything serious with data, then you should be taking this seriously.

Fix the "You need permission" error in Azure Synapse Analytics with this guide, addressing its causes and solutions for Data Engineers/Developers.

Assign roles in Synapse Studio for Azure Synapse Analytics devs using Azure CLI. Accessible by Owners/Contributors of the resource.

Despite the subsequent media reporting, the loss of 16,000 Covid-19 test results at Public Health England wasn't caused by Excel. This post argues that a lack of an appropriate risk and mitigation analysis left the process exposed to human error, which ultimately led to the loss of data and inaccurate reporting. It describes a simple process that could have been applied to prevent the error, and how it will help if you're worried about ensuring quality or reducing risk in your own business, technology or data programmes.

Explore Azure Synapse Link for Cosmos DB's impact on 'full stack serverless', No-ETL, and pay-as-you-query analytics.

Wishing Azure Synapse Analytics had support for SQL notebooks? Fear not, it's easy to take advantage rich interactive notebooks for SQL Pools and SQL on Demand.

Tracking down unexpected allocations in a high-performance .NET parsing library.

Explore deploying Azure Synapse Analytics workspaces using ARM templates, a popular infrastructure deployment method for organizations.

Serverless data architectures enable leaner data insights and operations. How do you reap the rewards while avoiding the potential pitfalls?

I was recently invited on to Microsoft Mechanics to talk about the new on-demand SQL Serverless offering within Azure Synapse. If you have been following along with my previous blog posts you will know that we've been hard at work applying Azure Synapse against real customer workloads. In the video I take you through the service by solving a real-world IoT problem for one of our telco customers.

New Azure Synapse Analytics service offers SQL Serverless for on-demand data lake queries. We tested its potential as a Data Lake Analytics replacement.

Explore why Microsoft's new Spark offering in Azure Synapse Analytics is a game-changer for Azure Databricks investors.

Explore 5 key reasons to choose Azure Synapse Analytics for your cloud data needs, based on years of experience in driving customer outcomes.

Whilst "read/write XMLA endpoint" might seem like a technical mouthful, its addition to Power BI is a significant milestone in the strategy of bringing Power BI and Analysis Services closer together. As well as closing the gap between IT-managed workloads and self-service BI, it presents a number of new opportunities for Power BI developers in terms of tooling, process and integrations. This post highlights some of the key advantages of this new capability and what they mean for the Power BI developer.

Ensure Power BI report quality by connecting to tabular models, executing scenario-based specs, and validating data, business rules, and security.

Jess and Carmel recently gave a talk at Azure Oxford on Combatting illegal fishing with Machine Learning and Azure - for less than £10 / month. The recording of that talk is now available for viewing!The talk focuses on the recent work we completed with OceanMind. They run through how to construct a cloud-first architecture based on serverless and data analytics technologies and explore the important principles and challenges in designing this kind of solution. Finally, we see how the architecture we designed through this process not only provides all the benefits of the cloud (reliability, scalability, security), but because of the pay-as-you-go compute model, has a compute cost that we could barely believe!

Ensure reliable insights with endjin's automated quality gates for validating Power BI Dataflows in complex solutions.

Explore Azure Analysis Services for scalable analytics. Control costs via automation with Powershell & Azure DevOps in multi-environment setups.

Endjin's blog post details their project with OceanMind, using a serverless architecture and machine learning to detect illegal fishing.

Learn how to update Azure Analysis Services model schemas in .NET apps using AMO SDK for user-driven analysis.

Tour our OceanMind project using Azure Functions for real-time marine telemetry processing. Learn optimization techniques for serverless environments.

Learn to use Azure Analysis Services in custom apps via REST API in .NET for efficient async model refreshes.

Explore endjin's high-performance .NET AIS parser, developed for OceanMind, used for tracking commercial ships.

Explore endless possibilities with dynamic DAX queries in C# for Azure Analysis Services integration in custom apps using the provided code samples.

The theme of this year's British Science Week (6 - 15 March 2020) is "Our Diverse Planet". We'll be getting involved by speaking to school children about the work we've been doing with Oxfordshire-based OceanMind (part of the Microsoft AI for Good programme) to help them combat illegal fishing, hopefully inspiring some of the next generation of data scientists!

Explore .NET querying methods for integrating Azure Analysis Services beyond data querying into dynamic UIs and APIs.

Learn to integrate Azure Analysis Services in apps by establishing server connections. Follow this guide with code samples for essential scenarios.

Explore Azure Analysis Services in custom apps using SDKs, PowerShell cmdlets & REST APIs. Learn to choose the right framework in this guide.

Explore Azure Analysis Services' versatility in bespoke analysis products & processes, and learn how it can unlock data insights beyond traditional BI.

Endjin attended Microsoft's AI for Good hackathon at the IET in London, highlighting the potential of tech to amplify good deeds.

In this blog we discuss building a secure data solution using Azure Data Lake. Data Lake has many features which enable fine grained security and data separation. It is also built on Azure Storage which enables us to take advantage of all of those features and means that ADLS is still a cost effective storage option!This post runs through some of the great features of ADLS and runs through an example of how we build our solutions using this technology!

In January 2020, Carmel is speaking about creating high performance geospatial algorithms in C# which can detect suspicious vessel activity, which is used to help alert law enforcement to illegal fishing. The input data is fed from Azure Data Lake Storage Gen 2, and converted into data projections optimised for high-performance computation. This code is then hosted in Azure Functions for cheap, consumption based processing.

Explore how ref struct types like Span<T> in C# enhance performance but pose async method challenges, and learn mitigation techniques.

Explore techniques for high-performance, low-allocation code in Azure Functions using C#, including data streaming, list preallocation, and Span&lt;T&gt;.

Learn to import/export notebooks in Databricks workspaces manually or programmatically, and transfer content between workspaces efficiently.

Machine learning often seems like a black box. This post walks through what's actually happening under the covers, in an attempt to de-mystify the process!Neural networks are built up of neurons. In a shallow neural network we have an input layer, a "hidden" layer of neurons, and an output layer. For deep learning, there is simply more hidden layers which allows for combining neuron's inputs and outputs to build up a more detailed picture.If you have an interest in Machine Learning and what is really happening, definitely give this a read (WARNING: Some algebra ahead...)!

Explore solutions for JsonDecodeError in Databricks CLI & Clusters. Learn how pre-built CLIs/SDKs simplify requests & authentication in REST APIs.

Explore data analysis & ETL with Databricks Notebooks on Azure. Utilize Spark's unified analytics engine for big data & ML, and integrate with ADF pipelines.

Snowflake is a cloud native data warehouse platform, that enabled data engineering, data science, data lakes, data sharing and data warehousing. Endjin are very excited to announce our partnership.

Mapping Data Flows are a relatively new feature of ADF. They allow you to visually build up complex data transformation sequences. This can aid in the streamlining of data manipulation and ETL processes, without the need to write any code! This post gives a brief introduction to the technology, and what this could enable!

Explore the lack of a native Azure Data Factory connector for Snowflake and discover alternatives for integration between these popular platforms.

When he joined endjin, Technical Fellow Ian sat down with founder Howard for a Q&A session. This was originally published on LinkedIn in 5 parts, but is republished here, in full. Ian talks about his path into computing, some highlights of his career, the evolution of the .NET ecosystem, AI, and the software engineering life.

If you're using the Gremlin API for Cosmos DB, you can now see how much each operation costs in Request Units.

This is the first blog in a series about dataflow. The series focuses on TPL dataflow, but this post gives an overview of dataflow as a whole.The crucial thing to understand when using dataflow is that the data is in control. In most conventional programming languages, the programmer determines how and when the code will run. In dataflow, it is the data that drives how the program executes. The movement of data controls the flow of the program.

Learn to use SQL Server's Python integration for efficient data handling. Eliminate clunky transfers and easily operationalize Python models/scripts.

Learn what types of things an apprentice gets up to at endjin a few months after joining. You could be learning about Neural Networks: algorithms which mimic the way biological systems process information. You could be attending Microsoft's Future Decoded conference, learning about Bots, CosmosDB, IoT and much more. Hopefully, you wouldn't be in hospital after a ruptured appendix!

We've been helping customers adopt Microsoft Azure since 2010, we have produced a lot of thought leadership to help people think about the steps required, the risk involved and how to plan a successful adoption.

Learn to build a Power BI dashboard using DirectQuery and ElasticQuery across multiple databases with Alice Waddicor.

Alice reflects on year 2, being given more responsibility, diving deeper into all aspects of software delivery, and the good habits she's been building.

What do machine learning and data science actually mean? This post digs into the detail behind the endjin approach to structured experimentation, arguing that the "science" is really all about following the process, allowing you to iterate to insights quickly when there are no guarantees of success.

We have produced an insightful booklet called "Embracing Disruption - Financial Services and the Microsoft Cloud" which examines the challenges and opportunities for the Financial Service Industry in the UK, through the lens of Microsoft Azure, Security, Privacy & Data Sovereignty, Data Ingestion, Transformation & Enrichment, Big Compute, Big Data, Insights & Visualisation, Infrastructure, Ops & Support, and the API Economy.

This post looks at what machine learning really is (and isn't), dispelling some of the myths and hype that have emerged as the interest in data science, predictive analytics and machine learning has grown. Without any hard guarantees of success, it argues that machine learning as a discipline is simply trial and error at scale – proving or disproving statistical scenarios through structured experimentation.