Microsoft Fabric
Next generation service lowering barriers to adoption and enabling democratisation of data & analytics
-
Pros
- New generation built on top of proven products (Power BI, Synapse)
- Easy to get started - SaaS model accelerates time to getting underlying infrastructure in place
- Streamlines the end to end process of getting from raw data to actionable insights
- OneLake provides unified storage layer for all compute engines
- Embracing open standard (delta) for all data at rest, with optimisation through VORDER
- Single unified user experience
- Integration of OpenAI Co-Pilot into development experience
- Common data & analytics languages supported (Python, SQL)
- Embracing open industry standards such as delta lake
- Strong integration between services
- Introduction of new capabilities
- Opinionated "happy path" architectures to de-risk and accelerate time to value
- New features to boost developer productivity and collaboration
- Strong alignment to Data Mesh principles
- Tight integration with Microsoft Purview for governance and management of data product lifecycle
-
Cons
- Loss of serverless compute as part of new capacity driven commercial model
- No support for C# in notebooks
- Mapping Data Flows are not supported
- Rapidly moving product, constant releases of new features
- Compute engines (Spark and SQL) are not fully unified
- User experience not intuitive for pro devs
Microsoft's 3rd generation data and analytics platform.
Microsoft Fabric is a third generation data and analytics platform, building on strong foundations that have been established by Azure Synapse Analytics and Power BI. It is a SaaS solution that further lowers barriers to adoption.
The following Wardley Map summarizes the evolution from Azure Synapse Analytics and Azure Data Lake (Gen 2) to Microsoft Fabric. A shift right driven by "SaaS-fication" and a shift up by making features more accessible to "citizen analysts":
A persona drive user experience
A unified user experience smooths the boundaries between data engineering, data science, analytics and business intelligence. Opening up opportunities to decentralise and democratise data and analytics at scale.
Familiar functionality carried over
Services that you know and love include Azure Data Factory (Synapse Pipelines), Spark Notebooks, Spark Jobs and the next generation of SQL compute engine. What is exciting is that many of the underlying platforms have been enhanced as part of the "SaaS-ification" process to remove some of the friction experienced in current platforms. For example time to spin up Spark clusters is reduced from minutes to seconds.
The following animation loop illustrates how Microsoft Fabric is changing the landscape by evolving and integrating data and analytics features that we are familiar with in platforms such as Azure Synapse Analytics and Power BI:
Some capabilities are not being carried over
However it is disappointing to see that some Synapse capabilities such as Mapping Dataflows are not supported by Microsoft Fabric. There is also no automated upgrade path to move workloads from Synapse to Fabric. See our Synapse verus Fabric: A Side by Side Comparison for a detailed comparison of Synapse versus Fabric, and some initial insight into migration options.
A new user interface that may take getting used to
Fabric makes it possible to explore data, run experiments, develop pipelines, publish curated data, run analytics, build interactive reports and operationalize solutions within a single web-based UI. Those who are familiar with Synapse Studio will likely take some time to adjust to the Microsoft Fabric user experience. It organises artefacts in a different way based on concepts such as personas, workspaces and domains.
Notebooks on steroids!
We are particularly excited about the enhanced Notebook experience. It feels intuitive, modern and more responsive. It builds on the proven notebook experience for documenting your experiments and orchestrating data pipelines. Adding new features such as comments (similar to Microsoft Word), co-editing and the ability to consume files and managed tables from the Lakehouse by dragging and dropping them into a notebook cell.
"Saas-ification" lowers boundaries to adoption
Setting up a new Microsoft Fabric workspace is a breeze because it is SaaS. Therefore security and sign-on is seamless. Users are able to create a Lakehouses (which can be thought of as "Azure Data Lake Storage as a Service") easily ingest (or consume data without moving it using the new shortcut feature), work with data using the array of different tools and publish data and insights.
By providing a tightly integrated, self-serve data platform we envisage that this will enable the evolution of the wider "data value chain" as illustrated in the Wardley Map below. Here you can see how the evolution of the underlying platform will empower organisations to be more data driven:
Swiss Army knife
Microsoft Fabric is a vast product with a diverse set of features that can be harnessed to meet your specific requirements.
It can often be daunting to choose the specific tools in the Swiss Army knife that are right for each specific workload you are seeking to adopt.
We recommend that you take the time to analyse requirements (both functional and non-functional), using this to frame a considered evaluation to choose the Fabric features and architecture that is right for you.
AI driven productivity
Another very interesting feature is that Microsoft Fabric is also seeking to leverage Microsoft's investment in OpenAI, with co-pilot "prompt engineering" allowing users to describe their requirements and for the technical artefacts (Python, DAX and Power BI visuals) to be generated automatically.
Our experience so far with Copilot is mixed. We find that it is good at code generation (Python, DAX) but still needs a pro dev in the loop to validate the output. Using Copilot for generation of insights and visuals over your data is not as compelling as we might have thought. It often takes as much effort to write the prompt to get an acceptable result as it would for an experienced analyst to create this through traditional methods.
Also, do not under-estimate the importance of good architecture at all levels in the stack to enable the AI to perform in a reliable way. For example, only by constructing a robust semantic model which applies Kimble principles to the data model, tables and columns that are clearly named (ideally with descriptions and synonyms) and well architected DAX measures will the AI have a chance of producing reliable insights. This points to the need to have professionals in the loop before unleashing Copilot.
DataOps
Microsoft Fabric will cover your base data, analytics and AI needs: providing all of the tools you need to ingest, process, transform and harvest value from data. But it does not provide all of the capabilities you need for DataOps. To implement enterprise scale, production workloads that can be governed and sustained over the long term you will still need to integrate Microsoft Fabric with other services including products such as:
- Microsoft Purview - to provide overarching repository to discover, register, organise and govern data assets.
- Azure Key Vault - for managing secrets.
- Azure Application Insights and Log Analytics - to fulfil observability requirements.
- Azure DevOps and/or GitHub - to provide source control and CI/CD pipelines.
- Azure Machine Learning - for more complex ML and AI requirements.
- Package Services Such as PyPI - there are also a wide array of open source packages that can be integrated into Fabric. Care is required to select and then manage the ongoing evolution of any package that is chosen.
- Gateways for on premise connectivity.
Some of these services are straightforward to deploy and integrate. Others will require more effort to implement.
Seek out Microsoft partners who have experience backed up by architecture blueprints and IP that can de-risk and fast track delivery of you DataOps needs.
Huge potential
We love that Microsoft Fabric is seeking to accelerate time to value by reducing the cognitive load that has typically come with earlier generation platforms. No longer is there a need to deploy, integrate and manage Azure Resources to establish a base platform. Furthermore, Microsoft overlays an opinionated view of how to architect common workloads, so you don't have to start with a blank sheet of paper. New concepts such as workspaces and domains will provide the layers of abstraction over artifacts to govern, and protect your analytics ecosystem.
It is clear Microsoft Fabric represents a significant step forward for organisations that are seeking to become more data driven.
Constant evolution
Microsoft continue to put significant investment into the platform, new features are constantly being added. This is one of the benefits of choosing to adopt the platform, but it can also be daunting for the team to stay on top of new features and to asses which are relevant for your organisation.
We recommend that you set aside time for your team to continuously evaluate, learn and adopt new features where they add value.
Organisational readiness
Adoption of Microsoft Fabric should not be seen purely as a new technology platform, it's full value can only be realised within the wider context of your organisation scaling up its data capabilities not just through traditional IT or data teams but as an organisation wide initiative.
For that reason Microsoft Fabric should be approached as a socio-technical endeavour. In other words, successful adoption of Microsoft Fabric will be as much about changing hearts and minds, as it is about delivering technology. Read our decision makers series of blogs for more insights into how to think about Fabric as a strategic enabler for your organisation.
Read our blog posts about Microsoft Fabric
What is a Data Lakehouse?
DuckDB in Practice: Enterprise Integration and Architectural Patterns
