Microsoft Fabric ML Tutorial Part 5: Observability
Improve the observability of your data engineering processes using readily available features of Microsoft Fabric and Azure.
About this talk
Course
In part 5 of this course Barry Smart, Director of Data and AI, walks through a demo showing how to improve observability of your data engineering processes using readily available technology and platforms in with Microsoft Fabric and Azure.
Barry begins the video by explaining that one of the pitfalls of using Fabric Notebooks in an operational setting is that it can impede observability of what is going on within the notebook. To address this he shows how standard Python Logging, combined with Azure Application Insights can be used to emit telemetry from key points in your notebook and captured centrally. We demonstrate this approach with a create_logger function, showcasing its application in a Fabric notebook and the benefits of centralizing logs.
This episode helps ensure continuous monitoring and rapid issue detection for optimal data product performance. Application Insights then provides to the tools ot enable you to proactively monitor the behaviour of your entire data estate to sustain value over the long term. It also provides a valuable source of troubleshooting information when things do go wrong to help you quickly diagnose and resolve issues.
Stay tuned for our next episode on deploying resources through infrastructure as code.
Chapters:
- 00:00 Series Recap
- 00:34 Focus on Observability
- 02:33 Implementing Logging in Fabric
- 04:01 Demo: Logging in Action
- 05:26 Monitoring with Application Insights
- 06:02 Conclusion and Next Steps
- 08:45 Final Thoughts and Next Steps
From Descriptive to Predictive Analytics with Microsoft Fabric:
- Part 1 - Overview
- Part 2 - Data Validation with Great Expectations
- Part 3 - Testing Notebooks
- Part 4 - Task Flows
- Part 5 - Observability
Microsoft Fabric End to End Demo Series:
- Part 1 - Lakehouse & Medallion Architecture
- Part 2 - Plan and Architect a Data Project
- Part 3 - Ingest Data
- Part 4 - Creating a shortcut to ADLS Gen2 in Fabric
- Part 5 - Local OneLake Tools
- Part 6 - Role of the Silver Layer in the Medallion Architecture
- Part 7 - Processing Bronze to Silver using Fabric Notebooks
- Part 8 - Good Notebook Development Practices
Microsoft Fabric First Impressions:
Decision Maker's Guide to Microsoft Fabric
and find all the rest of our content here.