How to ensure quality in your data insights
In this session from SQLBits 2024, learn how to improve your data quality by implementing testing strategies for notebooks, data pipelines, and Power BI Reports.
About this talk
SQLBits 2024
In this comprehensive session, discover how to ensure accuracy and reliability in your data projects by implementing testing methodologies for notebooks, pipelines, and Power BI reports.
James Broome, Director of Engineering, explores real-world examples from dozens of endjin's data & analytics projects; teaching techniques like unit testing in Fabric Notebooks, snapshot testing for ETL processes in Synapse Pipelines, and user acceptance testing for Power BI models. Learn to avoid common pitfalls and enhance your data practices with effective testing strategies.
Chapters
- 00:00 Introduction and Context
- 00:16 The Importance of Accurate Data Reporting
- 01:53 Challenges in Data Testing
- 03:40 Common Excuses for Lack of Testing
- 05:29 Levels of Software Testing
- 08:23 Unit Testing with Fabric Notebooks
- 11:28 Refactoring for Testability
- 19:09 Integration Testing with Pipelines
- 22:45 Testing Pipeline Behaviour with Valid and Invalid Data
- 23:40 Running and Monitoring Pipeline Tests
- 25:21 Snapshot Testing for Data Validation
- 31:05 User Acceptance Testing in Power BI
- 33:51 Creating Executable Specifications with Gherkin
- 34:40 Building Dynamic DAX Queries for Testing
- 37:18 Testing Power BI Reports Locally and in the Cloud
- 43:17 Conclusion and Final Thoughts
About the presenter
James Broome
Director of Engineering
