Browse our archives by topic…
Azure Synapse Analytics
Microsoft Fabric: Announced
Microsoft Fabric extends the promise of Azure Synapse integration to all analytics workloads from the data engineer to the business knowledge worker. It brings together Power BI, Data Factory, and the Data Lake, on a new generation of the Synapse data infrastructure. Delivered as a unified SaaS offering, it aims to reduce cost and time to value, while enabling new "citizen data science" capabilities. Check out all the resources from the endjin team collated in this post.
What is OneLake?
OneLake is the core of Microsoft Fabric - it is essentially the storage account for all of your data utilized within Fabric, be that within Azure or from another cloud (through Shortcuts). As the name suggests, it is a single, logical data lake backing more or less all of your Fabric workloads. For fans of analogies: you'll have probably heard Microsoft use the tag-line "OneLake is to data what OneDrive is to files/documents". Read this blog to dive deeper into OneLake and learn about why you should be excited.
Azure Synapse Analytics versus Microsoft Fabric: A Side by Side Comparison
In this post we compare Azure Synapse Analytics with Microsoft Fabric to understand how features map from Synapse to Fabric.
Intro to Microsoft Fabric
Microsoft Fabric is a unified data & analytics platform that brings together, and improves upon, Microsoft's existing suite of data products. It can be thought of as the third generation of Microsoft data platforms, where first generation (e.g. HDInsight, SQL Data Warehouse) were somewhat isolated takes on traditional data products; second generation was Azure Synapse Analytics, which integrated platforms at a UX level but still felt a little disjointed at the data level; and now we have Microsoft Fabric which builds upon the Synapse "unification" vision, with a particular focus on enabling deep data-level interoperability. In this post we introduce its core offerings, as well as weigh up the pros and cons of this new product.
Notebooks in Azure Synapse Analytics
This blog post explores interactive notebooks in Azure Synapse Analytics. A Synapse Notebook is a powerful data science tool that can be utilised in a variety of contexts including exploratory data analysis, data cleaning and transformation, data visualisation, statistical modeling and machine learning.
Working with JSON in Pyspark
This blog shows how to transform a document based format to a tabular format whilst managing all the nested properties
Customizing Lake Databases in Azure Synapse Analytics
Great, so I've configured my Lake Database in Azure Synapse Analytics. But since I'm using parquet-backed files, my column names aren't very user-friendly. I also have these calculated columns incorporating business logic that I'd like to query on the fly rather than persist them to backing data. I also want to give specific end-users access to this database and provide them with pre-defined reporting queries to get them up and running as quickly as possible. How can I do this? Enter Custom Objects in Lake Databases - now you can create VIEWs, Stored Procedures, USERs (amongst other objects) in what used to be a read-only database. This article explores the customization options and how it can help you organize your reporting data in Azure Synapse Analytics
How to create a semantic model using Synapse Analytics Database Templates
In this second blog in the series, we put the newly released Azure Synapse Analytics Database Templates into action by exploring the different methods that are available to create a semantic model.
What is a Lake Database in Azure Synapse Analytics?
Lake Databases in Azure Synapse Analytics are just great. If you're starting on a new Synapse Analytics project, chances are you can benefit from Lake Databases. Whether you need to analyze business data from Dataverse, share your Spark tables of data with SQL Serverless, or use Database Templates to visually design and define your standardized/semantic analytical data model, there's a use-case for you. In this blog we'll explore what Lake Databases are, why they're useful, and what you need to watch-out for when using them.
Sharing access to synchronized Shared Metadata Model objects in Azure Synapse Analytics
The "Shared Metadata Model" is a powerful feature within Synapse Analytics that synchronizes Spark database objects with SQL Serverless. This article describes how to give non-admin users access to these synchronized objects in a least-privileged manner.
What is the Shared Metadata Model in Azure Synapse Analytics, and why should I use it?
A lesser known feature of Azure Synapse is the "Shared Metadata Model". Synapse has the capability to automatically synchronize tables created via Synapse Spark with objects you can query via the usual SQL Serverless endpoint, without any additional configuration. This article brings attention to this capability, highlighting the benefits and tradeoffs vs rolling your own SQL Serverless VIEWs.
Managing schemas in Azure Synapse SQL Serverless
SQL Serverless in Azure Synapse provides a structured way to query your data on-demand directly from your data lake. This consumption-based, flexible approach to data warehousing provides a compelling alternative to the traditional star-schema or RDBMS, but comes with it's own set of new challenges. This post explores some of the considerations around managing schemas in a serverless world, including an approach for ensuring that your data and schema remain in sync.
How to use Azure Synapse SQL Serverless to connect Data Lake and Power BI
TL;DR - Using Azure Synapse SQL Serverless, you can query Azure Data Lake and populate Power BI reports across multiple workspaces.
How to test Azure Synapse notebooks
Interactive Spark notebooks are an incredibly powerful tool for data exploration and experimentation. And in Azure Synapse, the time to (business) value is significantly decreased due to tight integration with Pipelines and monitoring tooling. But as with any software process, the need to validate business rules is important, as is ensuring that quality doesn't regress over time - especially so in such a collaborative and productive environment. This post looks at some simple ways to add testing to your Synapse Notebooks.
How to safely reference a nullable activity output in Azure Synapse Pipelines and Azure Data Factory
Did you know that there's a null-safe operator in Azure Data Factory's expression syntax? No? Well, now you do. Here's how you can safely reference an activity output that might not always exist.
How to trigger an Azure Synapse pipeline run from C#
A rundown of how to trigger an Azure Synapse pipeline run from C# code.
How Azure Synapse unifies your development experience
Modern analytics requires a multi-faceted approach, which can cause integration headaches. Azure Synapse's Swiss army knife approach can remove a lot of friction.
How to fix the "You need permission to access workspace..." error in Azure Synapse Analytics
Data Engineers/Developers want to get access to Azure Synapse Analytics as quickly as possible to start designing and creating their data solutions. Being denied access to Synapse Studio can be frustrating and slows matters down. This article will address the "You need permission to access workspace..." error, discuss what causes it, and describe how to fix it.
How to use the Azure CLI to manage access to Synapse Studio
Azure Synapse Analytics developers need to be assigned a role within Synapse Studio in order to access the GUI. The Azure CLI provides one way of programmatically achieving this, which can be done by any Owner or Contributor of the Azure Synapse Analytics resource.
Does Azure Synapse Link redefine the meaning of full stack serverless?
Azure Synapse Link for Cosmos DB is a game-changing piece in the Synapse suite of services - extending the support for SQL on Demand to enable querying over the Cosmos DB Analytical Store. This post explores whether the term 'full stack serverless' should now be extended to cover No-ETL and pay-as-you-query analytics, alongside serverless application architectures.
How to use SQL Notebooks to access Azure Synapse SQL Pools & SQL on demand
Wishing Azure Synapse Analytics had support for SQL notebooks? Fear not, it's easy to take advantage rich interactive notebooks for SQL Pools and SQL on Demand.
Deploy an Azure Synapse Analytics workspace using an ARM Template
Azure Synapse Analytics is Microsoft's new unified cloud analytics platform, which will surely be playing a big part in many organizations' technology stacks in the near future. For many organizations, Azure Resource Manager (ARM) templates are the infrastructure deployment method of choice. This blog explains how to deploy an Azure Synapse Analytics workspace using an ARM template.
Azure Synapse Analytics: How serverless is replacing the data warehouse
Serverless data architectures enable leaner data insights and operations. How do you reap the rewards while avoiding the potential pitfalls?
Talking about Azure Synapse on Microsoft Mechanics!
I was recently invited on to Microsoft Mechanics to talk about the new on-demand SQL Serverless offering within Azure Synapse. If you have been following along with my previous blog posts you will know that we've been hard at work applying Azure Synapse against real customer workloads. In the video I take you through the service by solving a real-world IoT problem for one of our telco customers.
Benchmarking Azure Synapse Analytics - SQL Serverless, using Polyglot Notebooks
There is a new service in town that promises to transform the way you query the contents of your data lake. Azure Synapse Analytics comes with a new offering called SQL Serverless allowing you to query your data on-demand with no need for pre-provisioned resources.When we heard about the new service we were keen to get involved, so for the last 10 months we've been working with the SQL Serverless product group to provide feedback on the service and to help ensure it meets our customers needs. During this time we've put it through it's paces by implementing a range of real-world use cases. We were particularly interested to see how it stacked up as a replacement for Data Lake Analytics, where to date there has been no clear and easy migration path.
Azure Synapse for C# Developers: 5 things you need to know
Did you know that Azure Synapse has great support for .NET and #csharp? Learning new languages is often a barrier to digital transformation, being able to use existing people, skills, tools and engineering disciplines can be a massive advantage.
Does Azure Synapse Analytics spell the end for Azure Databricks?
Have you or are you about to invest in Azure Databricks? If so, the new Spark offering in Azure Synapse Analytics is likely to have grabbed your attention and rightly so. Why is Microsoft putting yet another Spark offering on the table and what does it mean for you?
5 Reasons why Azure Synapse Analytics should be on your roadmap
For years we have been building modern cloud data solutions on Azure and helping our customers transform their use of data to drive outcomes. Here are 5 reasons why Azure Synapse Analytics might just be the service that we have been crying out for.
Long Running Functions in Azure Data Factory
While on first inspection Azure Function look like a good candidate for long running operations as they can run for 10 minutes on a Consumption plan, HTTP Triggers only run for 230 seconds because of a limitation with the Azure Load Balancer. In this blog post Jess Panni demonstrates how Durable Functions can be used instead.