https://endjin.com/what-we-think/talks/reactive-extensions-for-dotnet-rxdotnet-v7-and-futures https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/rx-dotnet-v7-and-futures.jpg Reactive Extensions for .NET - Rx.NET v7 and Futures Ian Griffiths, Technical Fellow at endjin, .NET MVP, and author of Programming C# (O'Reilly), returns to On .NET Live to demo Rx.NET with live ship-tracking data from Norway's AIS network and walk through the major changes coming to the Reactive Extensions ecosystem in v7. In this episode: 🚢 Live demo — streaming real-time vessel data with Rx.NET and AIS.NET, using LINQ queries over observable sequences to join, group, and display ship positions on a WPF map 📦 System.Linq.Async → System.Linq.AsyncEnumerable in .NET 10 — how LINQ for IAsyncEnumerable moved from the Rx repo into the .NET runtime, and what that means for your projects ⚠️ Rx 7 Preview — unbundling WPF and Windows Forms support from System.Reactive to eliminate the 90MB binary bloat in self-contained deployments 🔮 Rx 8 and beyond — plans for browser WASM and Unity support, improved trimability, and the path to production-ready Async Rx Links & resources: Rx .NET repo FREE Ebook - Introduction to Rx .NET Rx .NET 7 Package Split ADR System.Linq.AsyncEnumerable in .NET 10 AIS.NET How .NET 10.0 boosted AIS.NET performance by 7% endjin blog ReactiveUI https://www.youtube.com/embed/8OAvFiczZ2k?t=187 3570 2026-02-25T06:30:00Z yes no https://endjin.com/what-we-think/talks/reactive-extensions-for-dotnet-status-and-plans-for-dotnet-10 https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/dotnetconf-2025-rx-dotnet-status-and-plans.jpg Reactive Extensions for .NET Status and Plans for .NET 10 Ian Griffiths, Technical Fellow at endjin, shares the latest updates on the Reactive Extensions for .NET (AKA ReactiveX AKA Rx.NET). Learn about the new features in Rx 6.1, what .NET 10 means for the project, and the significant packaging changes coming in Rx v7.0 that finally solve the long-standing deployment bloat issue. In this talk: Rx 6.1 New Features — DisposeWith operator for fluent CompositeDisposable usage, new TakeUntil overload with cancellation token support, and ResetExceptionDispatchState operator The Bloat Problem Explained — Why self-contained Windows deployments were pulling in 90MB of unnecessary WPF and Windows Forms assemblies Rx 7 Preview — How the new packaging model fixes bloat while maintaining source and binary compatibility Community Contributions — Features from Chris Pullman (ReactiveUI), Neils Berger, Daniel Weber, and Adam Jones Async Rx .NET — Status update and plans for a non-alpha release https://www.youtube.com/embed/y7Ks_bwSHUg 1590 2025-12-24T18:30:00Z yes no https://endjin.com/what-we-think/talks/csharp-14-new-feature-implicit-span-conversions https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/csharp-14-new-feature-implicit-span-conversions.jpg C# 14 New Feature: Implicit Span Conversions In this talk, Ian Griffiths dives into the new implicit conversions introduced in C# 14, designed to make span types more natural to use. He discusses how this change enhances performance, simplifies method signatures, and enables more powerful extension methods. However, Ian also warns about potential compatibility issues with older libraries and provides advice for library authors. He concludes with technical examples and solutions to common problems caused by these new features. Essential viewing for C# developers looking to leverage spans in their code more effectively. 00:00 Introduction to Implicit Conversions in C# 14 01:17 Understanding Span Types and Their Benefits 02:35 Practical Examples of Using Spans 05:04 Limitations and Issues in C# 13 08:05 Improvements in C# 14 12:42 New Implicit Span Conversions 19:16 Potential Issues with Older Libraries 23:33 Conclusion and Final Thoughts https://www.youtube.com/embed/C8yGHfk-puA 364 2025-12-11T06:30:00Z yes no https://endjin.com/what-we-think/talks/csharp-14-new-feature-field-backed-properties https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/csharp-14-new-features-field-backed-properties.jpg C# 14 New Feature: Field-Backed Properties In this talk, Ian Griffiths explains how C# 14's new field-backed properties feature can save you from metaphorically falling off a cliff when you need more flexibility beyond automatic properties' basic functionality. He demonstrates the use of this feature to customize property setters without losing the simplicity and support of automatic properties. By allowing you to refer to the compiler-generated field inside get or set methods, C# 14 reduces verbosity and maintains code clarity and organization. Learn how this small but impactful enhancement can improve your C# coding experience. 00:00 Introduction to C# 14's New Feature 00:30 Understanding Automatic Properties 01:11 Customizing Property Behavior 03:06 Introducing C# 14's New Syntax 04:21 Benefits of the New Feature 05:33 Conclusion https://www.youtube.com/embed/aAMQOBZA2bI 364 2025-12-03T00:00:00Z yes no https://endjin.com/what-we-think/talks/csharp-14-new-feature-script-directives https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/csharp-14-script-directives.jpg C# 14 New Feature: Script Directives C# 14 introduces new directives that transform C# into a true scripting language. In this video, Ian Griffiths explains how the .NET 10 SDK now lets you run a single C# source file directly—no project file required. What you'll learn: How to run C# files directly with dotnet run The shebang (#!) directive for Unix/Linux script execution The new #: directive for adding NuGet packages and build properties How these features build on the low-ceremony entry points introduced in C# 9 Contents: 00:00 Introduction to C# 14 Scripting Capabilities 00:10 Running C# Source Files with .NET 10 SDK 00:32 Simplifying C# Program Structure 01:47 Shebang Syntax for Unix Systems 04:05 Ignored Directives in C# 14 04:44 Using External Libraries in C# Scripts 07:38 Conclusion https://www.youtube.com/embed/WinOQSYxda8 1535 2025-11-28T06:30:00Z yes no https://endjin.com/what-we-think/talks/rxdotnet-v6-1-new-feature-resetexceptiondispatchstate https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/rx-dotnet-dispatch-state.png Rx.NET v6.1 New Feature: ResetExceptionDispatchState() In this video, Ian Griffiths introduces the new ResetExceptionDispatchState operator in Rx.NET 6.1 released in October 2025. He explains the peculiar behaviour of exception stack traces that led to the creation of this operator, following feedback from Adam Jones. The video delves into how exception state is managed in .NET and the specific issues that arise when exceptions are reused without being re-thrown. Ian demonstrates the problem with code examples and shows how the new operator resolves it. Full documentation is available at Introduction to Rx.NET. 00:00 Introduction to Rx.NET's New Feature 00:35 Background and Origin of the New Operator 02:35 Understanding Exception State in .NET 05:37 Demonstrating the Issue with Examples 12:31 Introducing the ResetExceptionDispatchState Operator 15:18 Conclusion and Further Resources https://www.youtube.com/embed/_4BVPQit6EM 985 2025-10-03T16:33:00Z yes no https://endjin.com/what-we-think/talks/rxdotnet-v6-1-new-feature-disposewith https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/rx-dotnet-dispose-with.png Rx.NET v6.1 New Feature: DisposeWith() In this episode, Ian Griffiths introduces the new DisposeWith method available in Rx.NET version 6.1, released in October 2025. He discusses the new functionalities, including a new TakeUntil() overload and the ResetExceptionDispatchState operator, with additional videos to cover these features. Ian explains that DisposeWith is a community contribution by Chris Pullman, designed to facilitate a fluent coding style by simplifying the disposal of multiple subscriptions. He demonstrates the method in a console application, comparing it with the traditional CompositeDisposable approach, and highlights its ease of use for handling observable subscriptions. Full documentation is available at Introduction to Rx.NET. 00:00 Introduction to Dispose Operator in Rx.NET 00:43 Community Contribution and Background 01:26 Purpose and Design of DisposeWith Method 01:47 Demo: DisposeWith Method in Action 02:11 CompositeDisposable vs DisposeWith 03:22 Fluent Style Development with DisposeWith 04:32 Conclusion and Further Resources https://www.youtube.com/embed/6wQVb8iyLFU 304 2025-10-03T16:32:00Z yes no https://endjin.com/what-we-think/talks/rxdotnet-v6-1-new-feature-takeuntil-cancellationtoken https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/rx-dotnet-cancellation-token.png Rx.NET v6.1 New Feature: TakeUntil(CancellationToken) In this video, Ian Griffiths introduces the new TakeUntil(CancellationToken) operator in Rx.NET 6.1, released in October 2025. He discusses the purpose and functionality of this operator, which allows users to stop an infinite source when a cancellation token is signalled. Ian acknowledges the contributions of community members Nils Aufschläger and Daniel Weber in shaping and developing this feature. Through a simple example using the Interval operator, Ian demonstrates how this new operator works and explains its benefits. Learn how to manage infinite sources effectively with the new TakeUntil(CancellationToken) operator in Rx.NET 6.1. Full documentation is available at Introduction to Rx.NET. 00:00 Introduction to Rx.NET 6.1 and New Features 00:47 Community Contributions and Design Evolution 01:27 Understanding the TakeUntil(CancellationToken) Operator 03:08 Practical Example: Using TakeUntil with Interval 06:00 Summary and Documentation https://www.youtube.com/embed/-ivFY_Hv89Y 398 2025-10-03T16:31:00Z yes no https://endjin.com/what-we-think/talks/introduction-to-technical-architecture https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/introduction-to-technical-architecture.jpg Introduction to Technical Architecture In this episode, Carmel delves into the realm of technical architecture, highlighting its importance in software development. What You'll Learn Carmel breaks down the often-confused world of architecture types, demonstrating: Enterprise vs Technical Architecture - understanding the strategic versus implementation layers The dual nature of software architecture - structure as a noun, vision as a verb Significant design decisions - identifying choices that will be costly to reverse Technical leadership principles - bridging business requirements and implementation Cross-cutting concerns - managing the relationships between system components Key Insights from This Episode Discover why technical architecture goes beyond just drawing diagrams: How architects decompose solutions into constituent parts whilst maintaining coherence Why platform choices, design patterns, and abstraction levels require careful consideration The critical role of architects in preventing project fragmentation Essential communication skills needed across stakeholder levels The balance between current needs and future scalability Who Should Watch This Whether you're an aspiring architect, a senior developer looking to step up, or a project manager wanting to understand technical decision-making better, this video provides the foundational knowledge you need. Carmel's clear explanations make complex architectural concepts accessible whilst maintaining technical rigour. About This Series This is the first episode in endjin's comprehensive Technical Architecture series. Over the coming weeks, we'll explore architectural drivers, patterns, anti-patterns, and real-world case studies from our extensive project portfolio. We help small teams achieve big things through considered architectural decisions. Chapters 00:00 Introduction to Technical Architecture 00:11 Types of Architecture 00:29 Focus on Technical Architecture 00:42 Simon Brown's Insights on Software Architecture 01:37 Role and Responsibilities of a Technical Architect 02:27 Conclusion and Next Steps https://www.youtube.com/embed/3U3vCBaYEJY 166 2025-08-18T05:30:00Z yes no https://endjin.com/what-we-think/talks/rxdotnet-packaging-plan-2025 https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/rx-net-packaging-plan-july-2025.jpg Rx.NET Packaging Plan 2025 Ian Griffiths provides an update on the state of Rx.NET since our last talk in June. Ian addresses the current choices around packaging Rx, focuses on the "package bloat" problem and discusses the strides we've made in addressing this issue. Ian explores the new "Rx Gauntlet" test suite designed to validate proposed solutions and highlight the use of automated tests and Power BI reports to identify and solve packaging issues. He also compares two primary design options for future Rx versions, weighing their advantages and potential risks. Finally, he seeks community feedback to guide our next steps in releasing a stable Rx.NET v7 version. You can find the detailed write up of this topic via the GitHub Discussion: Packaging Plans July 2025 #2211 Contents: 00:00 - Introduction and Update 00:12 - Addressing the Bloat Problem 01:34 - Evidence-Based Approach and Prototypes 02:09 - Introducing RX Gauntlet 02:56 - Testing and Results 06:54 - Power BI Report Insights 10:44 - Design Options and Future Plans 14:55 - Community Feedback and Next Steps This talk was given during endjin's internal weekly "show & tell" meeting, but has been edited for public consumption. https://www.youtube.com/embed/GSDspWHo0bo 925 2025-08-04T04:30:00Z yes no https://endjin.com/what-we-think/talks/medallion-architecture-in-practice https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/medallion-architecture-in-practice.jpg Medallion Architecture in Practice In this episode, Carmel Eve delves deeper into the connection between the medallion architecture and the semantic layer, exploring when data becomes truly production ready. Carmel recaps the three tiers of the medallion architecture: raw bronze, cleaned silver, and opinionated gold layers. The focus is placed on the semantic layer, which adds context and meaning to the gold layer data, making it useful for humans, machines, and AI. This semantic layer includes human or machine-friendly names, advanced data types, metadata for governance, and relationships between data objects. Carmel also discusses the environments for software development—development, testing, and production—and how this multi-environment system applies to the medallion architecture. Finally, we highlight the advantages of using this pattern, such as supporting multiple use cases, enhancing data lineage, and ensuring data's flexibility and security. Chapters 00:00 Introduction and Recap 00:06 Understanding the Semantic Layer 01:21 Defining the Semantic Layer 03:02 Data Production Readiness 03:13 Development, Testing, and Production Environments 04:43 Applying the Medallion Architecture 09:30 Conclusion and Key Takeaways https://www.youtube.com/embed/RHdGb0ZkI7c 411 2025-07-29T05:30:00Z yes no https://endjin.com/what-we-think/talks/introduction-to-the-medallion-architecture https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/introduction-to-the-medallion-architecture.jpg Introduction to the Medallion Architecture In this video, Carmel Eve provides a comprehensive introduction to the Medallion Architecture, a widely-adopted data design pattern for modern data platforms. What You'll Learn This video covers the three-tier architecture pattern that helps organizations structure their data lakes and lakehouses effectively: Bronze Tier: Raw data ingestion and historical archiving Silver Tier: Data transformation, validation, and cleaning Gold Tier: Business-ready data optimized for specific use cases Key Topics Covered The difference between ETL and ELT approaches How to transform various data formats (JSON, XML, CSV) into tabular structures Schema validation and data quality considerations When to combine data sources vs. keeping them separate Creating multiple data projections for different reporting needs Best practices for organizing data by domain and use case Who This Video Is For This video is ideal for: Data engineers building modern data platforms Data architects designing lakehouse architectures IT professionals working with Azure, Databricks, or similar platforms Anyone interested in understanding data design patterns Chapters 00:00 Introduction to Medallion Architecture 00:50 Understanding the Bronze Tier 01:46 Transforming Data in the Silver Tier 04:51 Creating the Gold Layer 06:43 Conclusion and Next Steps https://www.youtube.com/embed/xLvJEAnt3Vg 411 2025-07-17T15:00:00Z yes no https://endjin.com/what-we-think/talks/how-does-delta-lake-work https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/how-does-delta-lake-work.jpg How does Delta Lake work? In this video, Carmel Eve explores the game-changing technology that's finally made Data Lakehouses a practical reality for organisations worldwide. Following on from her introduction to Data Lakehouses, she dives deep into how OpenTable formats like Delta Lake, Apache Iceberg, and Apache Hudi have solved the performance challenges that previously limited adoption. What You'll Learn Carmel demonstrates how these innovative metadata layers bridge the gap between traditional data lakes and data warehouses, enabling: ACID transactions across multiple files - essential for data consistency Schema validation and enforcement - reject non-compliant data automatically Time travel and data versioning - create repeatable audit trails Unified batch and stream processing - support diverse workload patterns SQL-like querying performance - rival traditional databases whilst handling mixed data types Key Technical Insights Discover how OpenTable formats achieve massive performance improvements through: Intelligent indexing strategies that eliminate costly table scans Multi-tier caching mechanisms for frequently accessed data Statistical metadata collection for query optimization Z-ordering for multi-dimensional data clustering Predictive optimization capabilities in platforms like Databricks Unity Catalog Why This Matters For years, organisations have struggled to support both business analytics and data science workloads on the same platform. Carmel explains how this metadata revolution finally enables true convergence - allowing teams to work smarter, not harder, with their data infrastructure. Whether you're architecting a new data platform or optimising an existing one, understanding these OpenTable formats is crucial for modern data engineering success. Get in Touch Interested in implementing a Data Lakehouse architecture? Drop us a line at hello@endjin.com to discuss how these technologies can transform your data strategy. Chapters 00:00 Introduction to Data Lake Houses 00:37 Challenges with Traditional Data Lakes 01:33 Open Table Formats: A Game Changer 03:14 Performance Enhancements in Delta Lake 04:57 Advanced Data Management Techniques 06:10 Conclusion and Future Outlook https://www.youtube.com/embed/Z0BpxL561fM 395 2025-07-13T05:30:00Z yes no https://endjin.com/what-we-think/talks/what-is-a-data-lakehouse https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/what-is-a-data-lakehouse.jpg What is a Data Lakehouse? Ever wondered what exactly a Data Lakehouse is and why it's revolutionising how we interact with data? After returning from a career break in 2021, Carmel Eve discovered the data landscape had transformed beyond recognition. In this video, Carmel will guide you through the evolution of data architecture and explain why Lakehouses might be the solution you've been looking for. 🎯 What You'll Learn: The fundamental differences between Data Warehouses, Data Lakes, and Data Lakehouses Why ACID compliance matters for your data architecture How Microsoft Fabric leverages Delta Lake for optimised querying The real-world trade-offs between storage costs and performance Whether a Lakehouse architecture is right for your organisation 📚 In This Video: 00:00 Introduction to Data Lakehouse 01:06 Evolution of Data Storage: From Warehouses to Lakes 03:10 Challenges with Data Warehouses 04:11 Rise of Data Lakes 05:43 Limitations of Data Lakes 08:00 Introduction to Data Lakehouse Architecture 09:11 Microsoft Fabric and Future Prospects 10:58 Conclusion and Contact Information 💡 Key Takeaways: Data Lakehouses combine the best of both worlds - the low-cost, flexible storage of data lakes with the management features and performance of data warehouses. They support ACID transactions, schema enforcement, and direct BI querying whilst handling structured, unstructured, and semi-structured data. https://www.youtube.com/embed/3f295BOVy44 668 2025-07-02T05:30:00Z yes no https://endjin.com/what-we-think/talks/corvus-text-json-enhancing-json-data-handling-in-dotnet https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/corvus-text-json-enhanced-data-handling-in-dotnet.jpg Corvus.Text.Json - Enhancing JSON Data Handling in .NET In this technical talk, Matthew Adams explores Corvus.Text.Json, an experimental fork of System.Text.Json that introduces powerful new capabilities for working with JSON documents in .NET. This isn't just an incremental update to Corvus.JsonSchema - it's a fundamentally different approach that combines the best of immutable and mutable JSON handling. Key Topics Covered 🔧 Core Features Dual Model Support: Both immutable (for thread-safe operations) and mutable (for easier manipulation) JSON document handling Generic Element Types: Support for strongly-typed, code-generated JSON elements as first-class citizens JSON Schema Integration: Built-in schema matching and validation capabilities Performance Optimizations: Property maps for efficient lookups and zero-allocation operations 💡 Technical Highlights How the genericized document system works over custom element types The innovative workspace and builder pattern for safe document mutation Performance improvements through UTF-8 backing storage and optimized property lookups Deep equality comparisons and efficient document manipulation 🚀 Benefits Simplified API compared to existing Corvus.JsonSchema implementations Better performance than System.Text.Json's node-based approach Seamless integration between mutable and immutable representations Thread-safe document sharing with copy-on-write semantics Who Should Watch This talk is ideal for: .NET developers working with JSON processing Engineers interested in high-performance JSON manipulation Anyone using or considering Corvus.JsonSchema Developers looking for alternatives to System.Text.Json limitations Chapters 00:00 - Introduction to Corvus Text JSON 00:54 - Immutable Model Overview 02:16 - Mutable Model Request 03:35 - Experiment with System.Text.Json 09:07 - Schema Matching and Validation 11:59 - Deep Equality and Property Map 17:21 - Mutability and Document Builders 26:36 - Creating and Manipulating JSON Documents 38:51 - Handling Mutable and Immutable Elements 44:37 - Summary and Conclusion https://www.youtube.com/embed/oqQK1ltdrIo 843 2025-06-25T05:30:00Z yes no https://endjin.com/what-we-think/talks/ai-hallucinations-explained-why-its-not-a-bug-but-a-feature https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/ai-hallucinations-explained.jpg AI Hallucinations Explained: Why It's Not a Bug but a Feature In this episode, Ian Griffiths delves into the concept of AI hallucinations, explaining how it’s commonly misunderstood in the industry. AI hallucinations refer to the phenomenon where artificial intelligence systems generate seemingly plausible but factually incorrect outputs. Using examples from the legal and software development sectors, Ian argues that this behaviour should not be seen as a bug but rather as a feature that signifies AI's ability to contextualize language. AI, according to Griffiths, excels at creating conceptual frameworks to understand sentences, even when those frameworks describe events that never occurred. Mislabelling this as hallucination leads to unproductive attempts to correct a behaviour that is integral to AI's functionality. By accepting and working with this aspect of AI, systems can be designed more effectively to harness AI's true capabilities. 00:00 Understanding AI Hallucination 00:27 Examples of AI Hallucination 01:55 The Misconception of AI Hallucination 04:08 Human Perception vs. AI Reality 05:43 AI's Contextualization Power 06:16 The Jimmy White Example 14:35 AI's Cultural Knowledge 17:04 Practical Implications and Conclusion https://www.youtube.com/embed/N-UfNqGg6f8 1273 2025-06-18T05:30:00Z yes no https://endjin.com/what-we-think/talks/the-state-of-reactive-extensions-for-dotnet-in-2025 https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/the-state-of-reactive-extensions-for-dotnet-2025.png The State of Reactive Extensions for .NET in 2025 Rx.NET v6.0 was released in 2023. The free Introduction to Rx.NET book was released in 2024, but progress has stalled since then. In this hour long "state of the nation" talk, lead maintainer, Ian Griffiths, delves into the details, hidden complexities and the various options on the table, current progress and plans for Rx.NET v7.0. Contents: 00:06 - We've stalled 01:38 - The big issues 02:58 - Packaging 03:03 - Packaging problems 05:31 - Deep dive into packaging 08:48 - Packaging problems continued 09:16 - Deep dive into packaging continued 10:17 - Packaging problems continued 18:17 - Self-contained bloat on Windows 18:49 - Nuget.org example 21:36 - Self-contained bloat on Windows continued 24:12 - Difficulties fixing packaging problems 25:18 - Why a "clean break" doesn't work 27:04 - Rx packaging "gauntlet" 31:20 - I wouldn't be starting from here 33:35 - What are we doing about packaging? 35:08 - Option 1: workaround 36:39 - Option 2: split packages 38:13 - UWP 38:27 - UWP (aka uap10.0) 42:59 - What I'd like to do with UWP 43:54 - WASM 44:01 - WASM problems 46:37 - AsyncRx.NET 46:58 - AsyncRx.NET status 48:13 - System.Linq.Async 48:19 - Separation of System.Linq.Async 51:26 - TFM misperceptions 51:35 - Perception of non-support for .NET 8.0 51:57 - nuget.org example 52:54 - Perception of non-support for .NET 8.0 continued 54:04 - Why we will update the TFM anyway 54:07 - Backlog 54:19 - What about actual meaningful progress? 55:19 - Conclusion This talk was given during endjin's internal weekly "show & tell" meeting, but has been edited for public consumption. https://www.youtube.com/embed/SZpmj0fSfFI 3375 2025-06-02T00:00:00Z yes no https://endjin.com/what-we-think/talks/a-brief-introduction-to-streamlit-development https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/a-brief-introduction-to-streamlit-development.jpg A Brief Introduction to Streamlit Development In this video, we provide a brief introduction to Streamlit, an open-source Python library designed for quickly creating data visualizations with minimal code. We discuss its ability to facilitate interactive data exploration, making it a powerful tool for data science and machine learning projects. The video covers setting up a Streamlit project, importing necessary packages, and building an application using sample world cities data. Additionally, we demonstrate how to create and edit visualizations within the app, showcasing Streamlit's dynamic updating feature for real-time data manipulation. Join us to learn how Streamlit can enhance your data exploration capabilities. This is the second part in our series on Streamlit Development; to follow along you may need to watch part one: Simplify your Streamlit Python Development Experience with Dev Containers. 00:00 Introduction to Streamlit 00:13 Project Overview and Goals 00:39 Understanding Streamlit 01:07 Setting Up the Development Environment 01:49 Building the Streamlit Application 04:15 Creating Visualizations 04:41 Interactive Data Editing 05:58 Conclusion and Next Steps Useful links: Streamlit Docker Desktop WSL2 VS Code Dev Containers https://www.youtube.com/embed/1nZhPP8G1cY 384 2025-01-13T00:00:00Z yes no https://endjin.com/what-we-think/talks/simplify-your-streamlit-python-development-experience-with-dev-containers https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/simplify-your-streamlit-python-development-with-dev-containers.jpg Simplify your Streamlit Python Development Experience with Dev Containers Carmel Eve returned to endjin after a 3 year break to travel the world. She left a .NET Developer, and came back to discover a plethora of Python based data & analytics projects in-flight. Carmel has been recording her learning curve adapting to this new technology stack. In this video, she guides you through setting up and utilizing dev containers to create a controlled, repeatable, and isolated Python development environment. In this video you will learn how to install WSL and Docker Desktop on Windows, configure VS Code with Docker and DevContainers extensions, and initialize your Python project with Poetry. By following these steps, you can eliminate the hassle of managing dependencies and version conflicts across different projects. Perfect for any developer looking to streamline their Python development workflow. 00:00 Introduction to Local Development Containers 00:12 Challenges with Python Environment Management 00:41 Introduction to Dev Containers 01:28 Setting Up a Local Python Container 01:32 Installing WSL and Docker 02:55 Configuring VS Code for Dev Containers 04:31 Using Poetry for Dependency Management 06:51 Finalizing the Dev Container Setup 07:47 Conclusion and Next Steps Useful links: Introduction to Containers and Docker Streamlit Docker Desktop WSL2 VS Code Dev Containers Poetry Setting up multiple WSL distribution instances https://www.youtube.com/embed/NTmpVe5rhBg 482 2025-01-06T00:00:00Z yes no https://endjin.com/what-we-think/talks/an-introduction-to-corvus-jsonschema-code-generator-for-dotnet https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/jsonschema-code-generation-in-dotnet.jpg An Introduction to Corvus.JsonSchema Code Generator for .NET Matthew Adams takes you through a high level tour of Corvus.JsonSchema, an open-source, high-performance JSON schema validation and serialization library. The library generates C# code from JSON schema files, allowing for improved interaction and performance over traditional deserialization methods. The video demonstrates how to set up a .NET project, create JSON schema files, and utilize Corvus.JsonSchema's features like the newly added source generator, which eliminates the need for command-line tools by integrating code generation directly into the project. It also covers adding necessary NuGet packages and setting build actions for files. Practical examples include creating and validating a simple 'TestPerson' JSON schema file. The video concludes with a demonstration of generating structured data types based on schema constraints and highlights the benefits of a schema-first approach to code generation. https://www.youtube.com/embed/UsITbguqwgg 843 2024-12-18T00:00:00Z yes no https://endjin.com/what-we-think/talks/whats-up-with-half-maxvalue-in-dotnet https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/what-is-up-with-half-max-value-in-dotnet.jpg What's up with Half.MaxValue in .NET? In this episode of .NET Deep Dives, Ian Griffiths explores the peculiar behaviour of the half-precision floating point type. By using the simplicity of a C# program that prints an incorrect maximum value for this type, the video explains floating point representation intricacies, especially with 16-bit values. It delves into the concept of precision and range, illustrating the vast differences between single, double, and half-precision floats. The discussion clarifies .NET's formatting behaviour, which rounds half-precision values to avoid spurious precision in display, leading to seemingly wrong output. Endjin are proud to be a .NET Foundation Corporate Sponsor, as we are maintainers of Reactive Extensions for .NET (AKA ReactiveX AKA Rx.NET) which is one of the most well established and widely used open source .NET projects. https://www.youtube.com/embed/GZLAEUKX6Tk 707 2024-11-27T00:00:00Z yes no https://endjin.com/what-we-think/talks/csharp-source-generators-boost-productivity-in-the-rxdotnet-repo https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/dotnetconf-2024-source-generators-and-rx-net.jpg C# Source Generators Boost Productivity in the Rx.NET Repo Code generation has become increasingly important in recent versions of .NET, especially when using AOT compilation. But they can also play a role in boosting your power as a developer. This talk will show how we've been using source generators to extend the reach of the test suite in the Reactive Extensions for .NET. AsyncRx.NET was an experimental project with no tests, but source generators have made it possible to make the existing Rx.NET test suite work in this new project. Endjin are proud to be a .NET Foundation Corporate Sponsor, as we are maintainers of Reactive Extensions for .NET (AKA ReactiveX AKA Rx.NET) which is one of the most well established and widely used open source .NET projects. If you're interested in learning more about Rx.NET download the free Introduction to Rx.NET book. https://www.youtube.com/embed/WNbTp7iesEY 1535 2024-11-19T00:00:00Z yes no https://endjin.com/what-we-think/talks/data-mesh-and-microsoft-fabric-a-perfect-fit https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/does-it-work-data-mesh-and-microsoft-fabric.jpg Data Mesh and Microsoft Fabric - a perfect fit? In this video, Barry Smart, Director of Data & AI, examines how Microsoft Fabric can support a Data Mesh vision for data and analytics. As we transition to a digital age powered by data, we discuss how data professionals can help their organizations thrive. Discover the capabilities of Microsoft Fabric, its role in creating data products, and how it aligns with the core principles of Data Mesh. Learn about the advancements in technology, the integration of open-source tools, and the importance of a socio-technical approach to data. Don't miss insights on federated computational governance, and DataOps. Watch to understand how to drive value and innovation in your organization with Microsoft Fabric. The full transcript is available below. The talk contains the following chapters: 00:00 Introduction to Microsoft Fabric and Data Mesh 01:33 The Evolution of Data Platforms 02:57 Key Features of Microsoft Fabric 05:00 Introduction to Data Mesh 06:53 Principles of Data Mesh 09:36 Combining Microsoft Fabric and Data Mesh 11:00 Example Data Product in Fabric 12:58 Conclusion and Final Thoughts From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Part 3 - Testing Notebooks Part 4 - Task Flows Part 5 - Observability Microsoft Fabric End to End Demo: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices If you want to learn more about Fabric, take a look at some of our other content: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/M-KhoGRHZKU 884 2024-09-18T23:06:00Z yes no https://endjin.com/what-we-think/talks/from-descriptive-to-predictive-analytics-with-microsoft-fabric-part-5 https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/from-descriptive-to-predictive-analytics-with-fabric-05.jpg Microsoft Fabric Machine Learning Tutorial - Part 5 - Observability In part 5 of this course Barry Smart, Director of Data and AI, walks through a demo showing how to improve observability of your data engineering processes using readily available technology and platforms in with Microsoft Fabric and Azure. Barry begins the video by explaining that one of the pitfalls of using Fabric Notebooks in an operational setting is that it can impede observability of what is going on within the notebook. To address this he shows how standard Python Logging, combined with Azure Application Insights can be used to emit telemetry from key points in your notebook and captured centrally. We demonstrate this approach with a create_logger function, showcasing its application in a Fabric notebook and the benefits of centralizing logs. This episode helps ensure continuous monitoring and rapid issue detection for optimal data product performance. Application Insights then provides to the tools ot enable you to proactively monitor the behaviour of your entire data estate to sustain value over the long term. It also provides a valuable source of troubleshooting information when things do go wrong to help you quickly diagnose and resolve issues. Stay tuned for our next episode on deploying resources through infrastructure as code. Chapters: 00:00 Series Recap 00:34 Focus on Observability 02:33 Implementing Logging in Fabric 04:01 Demo: Logging in Action 05:26 Monitoring with Application Insights 06:02 Conclusion and Next Steps 08:45 Final Thoughts and Next Steps From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Part 3 - Testing Notebooks Part 4 - Task Flows Part 5 - Observability Microsoft Fabric End to End Demo Series: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices Microsoft Fabric First Impressions: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric A Tour Around Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/lVyLaxUn5e4 411 2024-09-11T23:06:30Z yes no https://endjin.com/what-we-think/talks/from-descriptive-to-predictive-analytics-with-microsoft-fabric-part-4 https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/from-descriptive-to-predictive-analytics-with-fabric-04.jpg Microsoft Fabric Machine Learning Tutorial - Part 4 - Task Flows In part 4 of this course Barry Smart, Director of Data and AI, walks through a demo showing how the new Task Flows feature in Microsoft Fabric can be applied. In this case, retro-fitting a suitable Task Flow to an existing project to show how it allows you to reinforce the specific architecture pattern you want to adopt and provide a means through which artefacts in a workspace can be better organised. Barry begins the video by explaining the architecture that is being adopted in the demo and the Medallion Architecture that has been adopted. He shows how a work around has been applied by naming the artefacts in the workspace with a numerical prefix to help sort and group them according to the stages in the medallion architecture. He then demonstrates how the new Task Flow feature in Microsoft Fabric can overcome this workaround. This forms part of an end to end demo of Microsoft Fabric that we will be providing as a series of videos over the coming weeks. Chapters: 00:00 Introduction to the New Task Flow Feature 00:16 Recap of the Titanic Diagnostic Analytics Series 00:37 Understanding the Medallion Architecture 01:53 Exploring the Workspace and Artifacts 03:08 Introducing Task Flows 04:09 Configuring Task Flows for Titanic Analytics 05:10 Organizing Artifacts with Task Flows 06:48 Utilizing Folders for Better Organization 08:45 Final Thoughts and Next Steps From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Part 3 - Testing Notebooks Part 4 - Task Flows Part 5 - Observability Microsoft Fabric End to End Demo Series: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices Microsoft Fabric First Impressions: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric A Tour Around Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/zkfSNwDaO-A 566 2024-09-05T23:06:30Z yes no https://endjin.com/what-we-think/talks/from-descriptive-to-predictive-analytics-with-microsoft-fabric-part-3 https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/from-descriptive-to-predictive-analytics-with-fabric-03.jpg Microsoft Fabric Machine Learning Tutorial - Part 3 - Testing Notebooks In part 3 of this course Barry Smart, Director of Data and AI, walks through a demo showing how to apply a test driven development approach to Microsoft Fabric Notebooks that will allow you to establish a set of tests that can be automated, whilst also driving code that is clean, extensible, re-usable and easy to understand. He will focus on the notebook which applies the data wrangling steps to "project to gold". Focusing in on the logic which is used to clean and enrich the passenger data for the Titanic. Barry splits this logic into 3 notebooks: The first defines the functionality as a series of discrete data wrangling functions, wrapped up in a Titanic Wrangler class. He uses the Pandas pipe method to chain these individual functions together to perform all of the tasks necessary to clean and enrich the passenger data. The second notebook tests this functionality, by using the Arrange, Act, Assert (AAA) pattern. The final notebook puts this functionality in use as part of the wider "project to gold" process which projects a fact table and a set of dimension tables to the Gold area of the lake in Delta format. Barry begins the video by explaining the architecture that is being adopted in the demo including Medallion Architecture and DataOps practices. He explains how these patterns have been applied to create a data product that provides Diagnostic Analytics of the Titanic data set. This forms part of an end to end demo of Microsoft Fabric that we will be providing as a series of videos over the coming weeks. Chapters: 00:00 Introduction and Video Overview 00:46 Project to Gold Pipeline 01:08 Benefits and Pitfalls of Notebooks 04:20 Addressing Notebook Pitfalls with DataOps 05:12 Test-Driven Development in Data Engineering 08:03 Implementing the Titanic Wrangler Class 09:17 Testing the Titanic Wrangler Class 10:52 Running the Code in Production 12:15 Conclusion and Next Steps From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Part 3 - Testing Notebooks Part 4 - Task Flows Part 5 - Observability Microsoft Fabric End to End Demo Series: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices Microsoft Fabric First Impressions: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric A Tour Around Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/Y5f8T_lf77o 773 2024-08-28T23:06:30Z yes no https://endjin.com/what-we-think/talks/how-to-ensure-quality-and-avoid-inaccuracies-in-your-data-insights https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/sqlbits-2024-how-to-ensure-quality-and-avoid-inaccuracies-in-your-data-insights.jpg How to ensure quality and avoid inaccuracies in your data insights In this comprehensive session, discover how to ensure accuracy and reliability in your data projects by implementing testing methodologies for notebooks, pipelines, and Power BI reports. James Broome, Director of Engineering, explores real-world examples from dozens of endjin's data & analytics projects; teaching techniques like unit testing in Fabric Notebooks, snapshot testing for ETL processes in Synapse Pipelines, and user acceptance testing for Power BI models. Learn to avoid common pitfalls and enhance your data practices with effective testing strategies. Chapters 00:00 Introduction and Context 00:16 The Importance of Accurate Data Reporting 01:53 Challenges in Data Testing 03:40 Common Excuses for Lack of Testing 05:29 Levels of Software Testing 08:23 Unit Testing with Fabric Notebooks 11:28 Refactoring for Testability 19:09 Integration Testing with Pipelines 22:45 Testing Pipeline Behaviour with Valid and Invalid Data 23:40 Running and Monitoring Pipeline Tests 25:21 Snapshot Testing for Data Validation 31:05 User Acceptance Testing in Power BI 33:51 Creating Executable Specifications with Gherkin 34:40 Building Dynamic DAX Queries for Testing 37:18 Testing Power BI Reports Locally and in the Cloud 43:17 Conclusion and Final Thoughts https://www.youtube.com/embed/jHiKzbXwxtw 2685 2024-08-22T00:00:00Z yes no https://endjin.com/what-we-think/talks/implementing-openchain-iso-iec-5230-and-iso-iec-18974 https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/webinar-implementing-openchain-isoiec-5230-at-endjin.webp Implementing OpenChain across endjin's InnerSource and Open Source estate Charlotte Gayton was invited by the OpenChain Project, an initiative of the Linux Foundation, to present a webinar about her 2 year journey with OpenChain. The webinar has two parts; the first is an overview of her industrial placement at endjin implementing the OpenChain ISO/IEC 5230 specification - the international standard for open source license compliance programs - across endjin's InnerSource and Open Source estate. The second focuses on her final year project on the OpenChain ISO/IEC 18974 specification - the industry standard for open source security assurance programs, building on the extensions to endjin's DevOps processes and tooling used to implement ISO/IEC 5230. The slides from the presentation are available below: You can also read Charlotte's final year dissertation: You can read more about Charlotte's journey with OpenChain in her blog series - What is OpenChain (ISO 5230)? https://www.youtube.com/embed/NdYYcAxC7Do 3161 2024-08-14T06:00:00Z yes no https://endjin.com/what-we-think/talks/compelling-data-storytelling-with-power-bi-global-brand-insights-20-years-of-financial-trends https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/compelling-data-storytelling-with-power-bi-20-years-of-financial-trends.jpg Compelling Data Storytelling with Power BI: Global Brand Insights - 20 Years of Financial Trends Explore the creative journey behind the Global Brand Insights Report on Power BI, showcased on the Power BI Data Stories Gallery. We crafted this report for educational purposes to highlight features in Power BI that allow you to work with data rich in imagery. In this case, we wanted to show how brand logos can be adopted in a Power BI report to improve accessibility and the drive an engaging user experience. In the report we work with financial data which has been sourced from Yahoo Finance. We explore visuals that work well with financial data that spans a time period of 20 years. This is aided by a custom date dimension which is used to aggregate financial results from the raw daily level, up to weekly, monthly, quarterly and annual levels. We explore the use of a 2D heat map to display high density information, showing how a RAG (read, amber, green) colour scheme can help your eye to isolate trends and anomalies in the data. We dynamically rank each company based on its share price in any year. This brings out some interesting techniques based on use of rank to explore relative performance of the brands over time. We adapt the Formula 1 lap chart visual for this purpose. We support the end to end user journey using a custom icon based navigation, custom tool tip and drill through. Chapters 00:00 Introduction to Endjin's Global Brand Insights Report 01:25 Data Sourcing and Preparation 02:40 Data Wrangling and Aggregation 04:45 Building the Power BI Report 05:44 Creating Interactive Visuals with DAX Measures 08:11 Exploring the Report Pages 10:48 Analyzing Trends and Rankings 13:25 Conclusion and Key Takeaways Useful Links Global Brand Insights: 20 Years of Financial Trends Power BI Data Stories Gallery: Learning From Disaster - Titanic Passenger Diagnostics Power BI Data Stories Gallery: World Bank Heath and Wealth Report Accessible Data Storytelling with Power BI: Design Concepts and Accessible Colours Data Storytelling with Power BI: The World Bank World Health and Wealth Report How to Build Navigation into Power BI How to develop an accessible colour palette for Power BI How to enable data teams with the design assets required for impactful data storytelling in Power BI How to Create Custom Buttons in Power BI How to Build a Branded Power BI Report Theme Generating custom themes in Power BI – A designer's perspective From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview of the Course Part 2 - Data Validation with Great Expectations https://www.youtube.com/embed/xBE4HcXG43k 963 2024-08-08T05:00:00Z yes no https://endjin.com/what-we-think/talks/microsoft-fabric-and-data-mesh-a-perfect-fit https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/sqlbits-2024-microsoft-fabric-and-data-mesh-a-perfect-fit.jpg Microsoft Fabric and Data Mesh - a perfect fit? Microsoft Fabric has been influenced by Data Mesh which is Zhamak Dehghani's thought leadership about how to "deliver data-driven value at scale". In this presentation, Barry will provide an overview of Data Mesh, how Microsoft Fabric measures up to Data Mesh principles and provide recommendations about how Fabric can be used to deliver a Data Mesh inspired data vision for data and analytics in your organisation.​ Chapters 00:00 Introduction to Microsoft Fabric and Data Mesh 00:33 Speaker's Background and Experience 01:52 The Digital Age and Role of Data Professionals 02:28 Overview of Microsoft Fabric 03:38 Introduction to Data Mesh 04:31 Principles of Data Mesh 05:57 Challenges and Solutions in Data Mesh Implementation 10:52 Data Mesh and Microsoft Fabric Integration 12:52 Strategies for Adopting Data Mesh with Fabric 14:55 Conclusion and Final Thoughts 17:22 Closing Remarks https://www.youtube.com/embed/NK9ARJ33zRQ 1169 2024-08-01T00:00:00Z yes no https://endjin.com/what-we-think/talks/how-to-actually-make-decisions-when-architecting-a-data-platform https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/sqlbits-2024-how-to-actually-make-decisions.jpg How To Actually Make Decisions When Architecting a Data Platform Are you drowning in a sea of architectural options for your new data platform? Are there people in your organisation with strong opinions on which tools should be used, and for what? Are you unsure how to prove/evidence your gut feeling as to how things should be built? Well - this session's for you! If you're in charge of architecting a new data platform, it can be difficult to make concrete decisions that pushes you closer to your end goal - building a successful solution. This session will introduce a tried-and-tested suite of strategies and tools that will help you efficiently navigate the unforgiving task of architecting a modern data platform. We'll take a look at artifacts such as ADRs (Architectural Decision Records), NFR (Non-functional requirements) documents, and discuss strategies inspired from the Agile methodology (such as failing fast) that can help you in your endeavour. Chapters 00:00 Introduction and Background 00:30 The Problem with Too Many Options 01:38 Understanding Platform Requirements 04:20 Organizational Factors and Conway's Law 07:53 Evaluating Products and Vendors 10:13 The Evaluation Process 15:05 Making the Final Decision 16:39 Closing Tips and Final Thoughts https://www.youtube.com/embed/5cpizhNX7bo 1164 2024-07-24T00:00:00Z yes no https://endjin.com/what-we-think/talks/compelling-data-storytelling-with-power-bi-titanic-survivors https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/compelling-data-storytelling-with-power-bi-titanic-survivors.jpg Compelling Data Storytelling with Power BI: Titanic Survivors Explore the creative journey behind the Titanic Passenger Diagnostic Report on Power BI, showcased at SQLBits 2024. Learn about our design approach, data sources from Kaggle, and how we structured the report like an interactive museum exhibit. Discover insights on demographics, survival rates, and how to create a visually engaging and informative data story. You can find the report on Power BI Data Stories Gallery. Chapters 00:00 Introduction to the Titanic Passenger Diagnostic Report 00:11 Data Source and Report Design Concept 00:26 Navigation and Creative Approach 01:19 Cover Page and Branding 01:51 Introduction Screen Overview 02:13 Demographics of Titanic Passengers 03:12 Who Survived: Analysis and Visuals 04:06 Conclusion and Presentation Tips Useful Links Power BI Data Stories Gallery: Learning From Disaster - Titanic Passenger Diagnostics Power BI Data Stories Gallery: World Bank Heath and Wealth Report Accessible Data Storytelling with Power BI: Design Concepts and Accessible Colours Data Storytelling with Power BI: The World Bank World Health and Wealth Report How to Build Navigation into Power BI How to develop an accessible colour palette for Power BI How to enable data teams with the design assets required for impactful data storytelling in Power BI How to Create Custom Buttons in Power BI How to Build a Branded Power BI Report Theme Generating custom themes in Power BI – A designer's perspective From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview of the Course Part 2 - Data Validation with Great Expectations https://www.youtube.com/embed/j2IF2T1yO90 489 2024-06-17T05:00:00Z yes no https://endjin.com/what-we-think/talks/10x-apache-spark-performance-improvement-in-microsoft-fabric https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/10x-apache-spark-performance-improvement-microsoft-fabric.jpg 10x Apache Spark performance improvement in Microsoft Fabric Boosting Apache Spark Performance with Small JSON Files in Microsoft Fabric. Learn how to achieve a 10x performance improvement when ingesting small JSON files in Apache Spark hosted on Microsoft Fabric. Ian Griffiths, Technical Fellow at endjin, shares insights and techniques to overcome Spark's challenges with numerous small files, including parallelizing file discovery and optimizing data loading. Follow along for detailed steps and tips to significantly enhance your Spark data processing workflows using Apache Spark in Microsoft Fabric. 00:00 Introduction to Performance Improvement in Apache Spark 00:20 Understanding the Problem with Small Files in Spark 00:38 Our Scenario: Performance Telemetry Collection 01:20 Initial Approach and Disappointment 01:40 Exploring the Root Cause 05:27 Parallelization: The Key to Performance Boost 08:51 Implementing the Solution in Spark 12:43 Conclusion: Balancing Complexity and Performance https://www.youtube.com/embed/nP0gogMbm6o 800 2024-06-11T23:06:00Z yes no https://endjin.com/what-we-think/talks/microsoft-fabric-good-notebook-development-practices https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/microsoft-fabric-e2e-08-good-notebook-development-practices.jpg Microsoft Fabric - Good Notebook Development Practices Notebooks can very easily become a large, unstructured dump of code with a chain of dependencies so convoluted that it becomes very difficult to track lineage throughout your transformations. With a few simple steps, you can turn notebooks into a well-structured, easy-to-follow repository for your code. In this video, we take a look at a few good practices when working with notebooks as a primary artifact type in your code-base. We'll look at things like using commentary within your notebooks, achieving modularization by structuring your logic into separate classes and functions, and defining "schemas as code" to be explicit about the data you're reading and writing. The full transcript is available below. The talk contains the following chapters: 00:00 Intro 00:12 Principles underpinning good notebook development 00:56 Methods to achieve good notebook development 02:27 Workspace artifact folder structure 05:27 Notebook structure demo - Commentary 06:19 Defining classes and functions 08:47 Consuming classes and functions - magic commands 09:36 Defining schemas as code 12:05 Consuming schemas when creating Delta tables 13:05 Round-up & Outro Useful links: 📖 Develop, execute, and manage Microsoft Fabric notebooks 📖 Create folders in workspaces (preview) Microsoft Fabric End to End Demo Series: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Part 3 - Testing Notebooks Part 4 - Task Flows Part 5 - Observability Microsoft Fabric First Impressions: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric A Tour Around Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/UyS6ZUgh-Wc 810 2024-05-09T05:30:00Z yes no https://endjin.com/what-we-think/talks/from-descriptive-to-predictive-analytics-with-microsoft-fabric-part-2 https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/from-descriptive-to-predictive-analytics-with-fabric-02.jpg Microsoft Fabric Machine Learning Tutorial - Part 2 - Data Validation with Great Expectations In part 2 of this course Barry Smart, Director of Data and AI, walks through a demo showing how you can use Microsoft Fabric to set up a "data contract" that establishes minimum data quality standards for data that is being processed by a data pipeline. He deliberately passes bad data into the pipeline to show how the process can be set up to "fail elegantly" by dropping the bad rows and continuing with only the good rows. Finally, he uses the new Teams pipeline activity in Fabric to show how you can send a message to the data stewards who are responsible for the data set, informing them that validation has failed, itemising the specific rows that failed and the validation errors that were generated in the body of the message. The demo uses the popular Titanic data set to show features in data engineering experience in Fabric, including Notebooks, Pipelines and the Lakehouse. It uses the popular Great Expectations Python package to establish the data contract and Microsoft's mssparkutils Python package to enable the exit value of the Notebook to be passed back to the Pipeline that has triggered it. Barry begins the video by explaining the architecture that is being adopted in the demo including Medallion Architecture and DataOps practices. He explains how these patterns have been applied to create a data product that provides Diagnostic Analytics of the Titanic data set. This forms part of an end to end demo of Microsoft Fabric that we will be providing as a series of videos over the coming weeks. Chapters: 00:12 Overview of the architecture 00:36 The focus for this video is processing data to Silver 00:55 The DataOps principles of data validation and alerting will be applied 02:19 Tour of the artefacts in the Microsoft Fabric workspace 02:56 Open the "Validation Location" notebook and viewing the contents 03:30 Inspect the reference data that is going to be validated by the notebook 05:14 Overview of the key stages in the notebook 05:39 Set up the notebook, using %run to establish utility functions 06:21 Set up a "data contract" using great expectations package 07:45 Load the data from the Bronze area of the lake 08:18 Validate the data by applying the "data contract" to it 08:36 Remove any bad records to create a clean data set 09:04 Write the clean data to the lakehouse in Delta format 09:52 Exit the notebook using mssparkutils to pass back validation results 10:53 Lineage is used to discover the pipeline that triggers it 11:01 Exploring the "Process to Silver" pipeline 11:35 An "If Condition" is configured to process the notebook exit value 11:56 A Teams pipeline activity is set up to notify users 12:51 Title and body of Teams message are populated with dynamic information 13:08 Word of caution about exposing sensitive information 13:28 What's in the next episode? From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Part 3 - Testing Notebooks Part 4 - Task Flows Part 5 - Observability Microsoft Fabric End to End Demo Series: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices Microsoft Fabric First Impressions: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric A Tour Around Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/-gxFodx0bNA 842 2024-05-07T00:06:30Z yes no https://endjin.com/what-we-think/talks/from-descriptive-to-predictive-analytics-with-microsoft-fabric-part-1 https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/from-descriptive-to-predictive-analytics-with-fabric-01.jpg Microsoft Fabric Machine Learning Tutorial - Part 1 - Overview of the Course In this video Barry Smart, Director of Data and AI, provides an overview of a new series providing an end-to-end demo of Microsoft Fabric focusing on a Predictive Analytics use case. The demo will use the popular Kaggle Titanic data set to show off features across both the data engineering and data science experiences in Fabric. This will include Notebooks, Pipelines, Semantic Link, MLflow (Experiments and Models), Direct Lake mode for Power BI and of course the Lakehouse. Barry explains some of the architecture principles that will be adopted during the demo including Medallion Architecture, DataOps practices and the Data Mesh principle of "data as a product". He explains how these patterns have been applied to shape the solution across data products that support Diagnostic Analytics and Predictive Analytics. Chapters: 00:14 Overview of Titanic data set and the features we will demo 00:44 The data landscape is evolving at pace 01:22 DataOps is a key focus for this demo 01:51 Adopting a product mindset 02:29 Goal of demo is to create a model that predicts survival on the Titanic 03:02 We will use the machine learning lifecycle 03:35 We deliver key stages of the machine learning lifecycle using two data products 04:55 Medallion architecture for the diagnostic analytics data product 06:41 MLflow enabled architecture for the predictive analytics data product 07:23 How data products can be chained together to build capability 07:59 What's in the next episode? From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Part 3 - Testing Notebooks Part 4 - Task Flows Part 5 - Observability Microsoft Fabric End to End Demo Series: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices Microsoft Fabric First Impressions: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric A Tour Around Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/uaRePHeqvQU 512 2024-04-28T23:06:30Z yes no https://endjin.com/what-we-think/talks/azure-functions-deep-dive https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/azure-functions-deep-dive.jpg Azure Functions Deep Dive In Episode #99 of the Real-World Serverless podcast, Yan Cui chats to Microsoft MVP, Ian Griffiths and take a deep dive into Azure Functions, how it works and how it differs (significantly) from AWS Lambda. https://www.youtube.com/embed/s_odkdt9qUU 3221 2024-03-31T00:00:00Z yes no https://endjin.com/what-we-think/talks/the-heart-of-reactive-extensions-for-dotnet https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/heart-of-rx.jpg The Heart of Reactive Extensions for .NET Reactive Extensions for .NET (Rx.NET) is profoundly useful, and developers often end up loving. Why is that? Endjin Technical Fellow Ian Griffiths explains how Rx.NET's deep roots in mathematical foundations make it both powerful and beautiful. To learn more about Rx.NET, check out the free book Introduction to Rx.NET 2nd Edition (2024). https://www.youtube.com/embed/9B4LRxUKpd4 318 2024-03-26T00:00:00Z yes no https://endjin.com/what-we-think/talks/microsoft-fabric-processing-bronze-to-silver-using-fabric-notebooks https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/microsoft-fabric-process-to-silver-using-notebooks.jpg Microsoft Fabric - Processing Bronze to Silver using Fabric Notebooks Notebooks in Fabric are a lot like notebook experiences in other tools you're probably already familiar with. They allow us to write code in a variety of languages and create a commentary alongside our code using interactive cells. Fabric notebooks also have built in integration with Lakehouses, and provide a built-in filesystem that can be used to store arbitrary files somehow associated with the notebook. Notebooks are our go-to tool for wrangling Bronze data into Silver, which tends to follow standard processing patterns. In this video we'll do a whirlwind tour of notebooks in Microsoft Fabric, and take a look at how our "Process to Silver" notebook takes shape. The full transcript is available below. The talk contains the following chapters: 00:00 Intro 00:10 Creating a blank notebook 01:04 Mounting a Lakehouse in a Fabric Notebook 02:05 Querying files in a Lakehouse from a Fabric Notebook 03:28 Query PySpark DataFrames directly with SQL 04:12 Built-in notebook filesystem 05:14 Walkthrough of "Process to Silver" notebook - reading data and transforming 08:18 Walkthrough of "Process to Silver" notebook - merging and writing data 10:03 Walkthrough of "Process to Silver" notebook - reading back written data 10:15 Roundup & Outro Useful links: 📖 Microsoft Fabric Notebooks 📖 Connect Lakehouses and Fabric Notebooks 📖 Built-in Notebook filesystem 📖 Delta Lake Merge Microsoft Fabric End to End Demo Series: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Part 3 - Testing Notebooks Part 4 - Task Flows Part 5 - Observability Microsoft Fabric First Impressions: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric A Tour Around Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/s_mHaLBlA94 373 2023-12-14T06:30:00Z yes no https://endjin.com/what-we-think/talks/microsoft-fabric-role-of-the-silver-layer-in-the-medallion-architecture https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/microsoft-fabric-role-of-the-silver-layer-in-the-medallion-architecture.jpg Microsoft Fabric - Role of the Silver Layer in the Medallion Architecture The Silver layer is where we apply standardization to our source datasets. This standardization aligns field names across sources, applies common data cleaning operations and organizes the data into a well known structure. The Silver layer, which is generally stored at full-fidelity (i.e. the granularity of the data is the same in the Silver layer as it is in the Bronze layer) provides the foundation for the Gold layer, where we may have many projections of the Silver data in different forms and at different granularities, depending on the downstream need. In this video we're going to talk about the role the Silver Lakehouse plays in the Medallion architecture, including what it's used for and whom it benefits. The full transcript is available below. The talk contains the following chapters: 00:00 Intro 00:13 Architecture diagram recap 01:18 Silver Lakehouse principles 04:10 Why is the Silver Lakehouse useful? 04:38 Who is the Silver Lakehouse for? 05:48 Roundup & Outro Useful links: 📖 Microsoft Fabric Medallion Architecture official docs Microsoft Fabric End to End Demo Series: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Part 3 - Testing Notebooks Part 4 - Task Flows Part 5 - Observability Microsoft Fabric First Impressions: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric A Tour Around Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/pCTl-nqDT_8 373 2023-12-14T05:30:00Z yes no https://endjin.com/what-we-think/talks/modernizing-reactive-extensions-for-dotnet https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/talk-dotnetconf-2023-modernizing-rxnet.jpg Modernizing Reactive Extensions for .NET Endjin are proud to be a .NET Foundation Corporate Sponsor, as we are maintainers of Reactive Extensions for .NET (AKA ReactiveX AKA Rx.NET) which is one of the most well established and widely used open source .NET projects. Learn how this project is evolving to take advantage of changes in .NET 8.0 https://www.youtube.com/embed/hLjqf4PmG9E 1723 2023-11-18T00:00:00Z yes no https://endjin.com/what-we-think/talks/high-performance-json-serialization-with-code-generation-on-csharp-11-and-dotnet-7-0 https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/jetbrains-dotnet-days-online-2023-high-performance-json-serialization-with-code-generation.jpg High Performance JSON Serialization With Code Generation on C# 11 and .NET 7.0 In this talk, you will see how features added to recent versions of C# can dramatically improve parsing and generation of JSON data compared with longer-established .NET JSON handling mechanisms. You will see how to use high-performance memory-efficient techniques, and also how code generators can reduce runtime overheads. This talk will explore the trade-off between maximizing performance and ease of use and will show you how to choose between the various options now available in modern .NET applications. https://www.youtube.com/embed/aTcD-axJBac 5201 2023-10-02T00:00:00Z yes no https://endjin.com/what-we-think/talks/microsoft-fabric-local-onelake-tools https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/microsoft-fabric-local-onelake-tools.jpg Microsoft Fabric - Local OneLake Tools Sometimes you need to be able to interact with your cloud data locally - maybe it's to troubleshoot/diagnose an issue, or just to do some analysis using your favourite local tools. Ideally you'd be able to browse your cloud data locally and avoid always having to download new copies of your files just to do this. Fortunately, there are a couple of tools that allow you to do just this with your OneLake data - namely the OneLake File Explorer and the trusty old Azure Storage Explorer. In this video we take a look at the two tools, and discuss some of the caveats you should be aware of when using them. The full transcript is available below. The talk contains the following chapters: 00:00 Intro 00:13 What is the OneLake File Explorer? 01:16 Recap: OneLake Lakehouse view 02:12 OneLake File Explorer Online Docs 02:33 Admin installation settings 03:12 Downloading OneLake File Explorer 03:30 Demo: Navigating OneLake File Explorer 04:24 Adding, deleting and syncing files 06:09 Gotcha 1 - "System" file access in OneLake File Explorer 07:20 Gotcha 2 - possible egress costs 09:55 What is Azure Storage Explorer? 10:32 Demo: Connecting to OneLake in Azure Storage Explorer 12:32 Navigating OneLake data in Azure Storage Explorer 13:57 Roundup & Outro Useful links: 📖 OneLake File Explorer docs 📖 Browsing OneLake with Azure Storage Explorer docs Microsoft Fabric End to End Demo Series: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Part 3 - Testing Notebooks Part 4 - Task Flows Part 5 - Observability Microsoft Fabric First Impressions: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric A Tour Around Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/DO779ZcAtTk 912 2023-08-21T05:30:00Z yes no https://endjin.com/what-we-think/talks/show-and-tell-a-brief-intro-to-tensors-and-gpt-with-torchsharp https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/show-and-tell-a-brief-intro-to-tensors-and-gpt-with-torchsharp.jpg Show & Tell - A Brief Intro to Tensors & GPT with TorchSharp Every Friday lunchtime, the whole of endjin get together for a "Show & Tell" session. These are essentially lightning talks on any subject; they can based on what someone has been working on that week, it can be a topic someone has been researching, it can be a useful tip or trick, it can be technical or non-technical; it just has to be something the speaker has found interesting, and wants to share with the whole company. This week Ian Griffiths, Technical Fellow, gave a brief overview of investigations into using TorchSharp to model Tensors, to use backpropagation to train Neural Networks, in order to understand the principals behind building your own Generative Pre-Trained Transformer (GPT). The Show & Tell contains the following chapters: 00:00 What is Torch / PyTorch / TorchSharp? 00:45 Representing numbers as Tensors 02:21 Tensors and the .NET Type System 04:07 Multiplying Arrays 05:16 Tensors and Neural Networks 06:23 GPUs and CUDA 08:35 How inputs affect output 12:21 What's backpropagation? 21:33 Building a GPT from scratch 24:07 Which libraries were used? Useful links: 🔦 TorchSharp 🔦 PyTorch 📺 Let's build GPT: from scratch, in code, spelled out. 🤖 Tensors 🤖 Backpropagation 🤖 Generative Pre-trained Transformer 📖 Polyglot Notebooks https://www.youtube.com/embed/KN9uiCHw9N0 1495 2023-08-09T23:06:00Z yes no https://endjin.com/what-we-think/talks/microsoft-fabric-creating-a-onelake-shortcut-to-adls-gen2 https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/microsoft-fabric-creating-a-onelake-shortcut-to-adls-gen2.jpg Microsoft Fabric - Creating a OneLake Shortcut to ADLS Gen2 Shortcuts in Microsoft Fabric enable zero-copy referencing of data across OneLake and clouds. With Shortcuts, you can get rid of your pipelines that are used to copy data into your storage account - instead, the data that's required for your Fabric operations is just retrieved as and when needed. In this video we use an Azure Data Lake Storage Gen2 (ADLS Gen2) account as the source, and configure a Shortcut in our Bronze Lakehouse so that the referenced data sits alongside our other raw data. The full transcript is available below. The talk contains the following chapters: 00:00 Intro 00:19 What are Shortcuts in Microsoft Fabric? 02:08 Demo - Browsing source data in ADLS Gen2 03:32 Where should I create my Shortcut? 04:46 Creating a Shortcut in Microsoft Fabric - Configuring connection 06:07 Configuring Shortcut settings 07:00 Browsing virtualized Shortcut data in Lakehouse View 08:04 Outro Useful links: 📖 OneLake shortcuts Microsoft Fabric End to End Demo Series: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Part 3 - Testing Notebooks Part 4 - Task Flows Part 5 - Observability Microsoft Fabric First Impressions: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric A Tour Around Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/11urrDAW-bU 522 2023-08-08T05:30:00Z yes no https://endjin.com/what-we-think/talks/decision-makers-guide-to-microsoft-fabric-the-pace-of-innovation https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/decision-makers-guide-to-microsoft-fabric-the-pace-of-innovation.jpg Decision Makers Guide to Microsoft Fabric - The Pace of Innovation Welcome to the final part in a new series of interviews with real-world Decision Makers (CTOs, CIOs, Heads of / Directors of Software Engineering, Data & Analytics) about how they manage their strategic roadmap and evaluate new technologies to simplify their portfolio, deliver better outcomes for stakeholders, or give them a competitive advantage. In a 3-part interview we talk to Tom Peplow about his assessment of Microsoft Fabric. TLDR; Tom Peplow, Principal & Senior Director Product Strategy at Milliman, chats with Ian Griffiths & Ed Freeman from endjin about how Microsoft Fabric is a disruptive technology. The insurance industry is slow moving, heavily regulated, and products have a long lifespan. Cloud platforms, on the other hand, are fast-paced and evolve quickly. How do you deal with these two opposing forces? Can you make good bets by betting on open source and open standards? Can you avoid vendor lock in, without the multi-cloud pitfall of building against the lowest common denominator? Or should you go all-in and bet on a single vendor and take advantage of their innovation investments? The talk contains the following chapters: 00:00 Introduction 00:30 Mismatched timescales? 01:30 Betting on open source 02:07 What will the market make of Fabric? 03:47 Is there lock-in with Fabric? 05:06 Shielding users from technology changes 06:25 When to bet on a platform 09:23 Multivendor hedging 09:51 Homogenization 11:09 Final thoughts From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Microsoft Fabric End to End Demo Series: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices Microsoft Fabric First Impressions: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric A Tour Around Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/bNegeu0Ytmo 711 2023-07-25T06:30:00Z yes no https://endjin.com/what-we-think/talks/decision-makers-guide-to-microsoft-fabric-microsoft-fabric-and-generative-ai https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/decision-makers-guide-to-microsoft-fabric-microsoft-fabric-and-generative-ai.jpg Decision Makers Guide to Microsoft Fabric - Microsoft Fabric and Generative AI Welcome to the second part in a new series of interviews with real-world Decision Makers (CTOs, CIOs, Heads of / Directors of Software Engineering, Data & Analytics) about how they manage their strategic roadmap and evaluate new technologies to simplify their portfolio, deliver better outcomes for stakeholders, or give them a competitive advantage. In a 3-part interview we talk to Tom Peplow about his assessment of Microsoft Fabric. In this part we talk about the relationship between Microsoft Fabric and AI. TLDR; Tom Peplow, Principal & Senior Director Product Strategy at Milliman, chats with Ian Griffiths & Ed Freeman from endjin about how AI could impact the insurance industry. Tom believes that AI is going to impact the future of business, and that it is up to IT leaders to help their organisations leverage the power of AI to better serve their customers. He discusses the relationship between Microsoft Fabric and AI, and how organisations can create a trusted AI environment where they can experiment with their own data to build AI-powered solutions. Tom believes that a good strategic IT department should be thinking about how to federate information across their organisation and with the outside world in order to train a Copilot on top of that information store. He believes that people inside an organisation will be 10x more effective at their jobs if they have access to AI-powered tools, and that there shouldn't be any fear around this. However, Tom is quick to point out that IT leaders need to figure out mechanisms for enabling businesses to grow quicker, leveraging tech, and not look for reasons to slow it down. He believes that it is important to build ethical AI frameworks inside of an organisation, in order to provide guidance and protection around internal intellectual property. Tom acknowledges that there is a skills gap when it comes to AI, but he is optimistic that this will get exponentially easier to harness as the technology matures. IT leaders need to be proactive in helping their organisations leverage its power to better serve their customers. The talk contains the following chapters: 00:00 Introduction 00:24 Empowering users and AI 01:32 Consuming corporate data with AI 04:05 Data isolation and protection in AI 06:48 Upskilling and AI From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Microsoft Fabric End to End Demo Series: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices Microsoft Fabric First Impressions: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric A Tour Around Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/8njSIrbG4Rg 560 2023-07-18T06:30:00Z yes no https://endjin.com/what-we-think/talks/modernizing-rxdotnet https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/on-dotnet-live-modernizing-rx.jpg Modernizing Rx.NET Reactive Extensions for .NET (Rx.NET) is a library for event-driven programming with a composable, declarative model. In this week's episode, Rx .NET Maintainer & Developer Technologies MVP, and endjin Technical Fellow Ian Griffiths, visits to show us the latest developments on this popular .NET project. https://www.youtube.com/embed/cDKb9isaDs8 3644 2023-07-18T00:00:00Z yes no https://endjin.com/what-we-think/talks/decision-makers-guide-to-microsoft-fabric-hedging-your-fabric-bet https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/decision-makers-guide-to-microsoft-fabric-hedging-your-fabric-bet.jpg Decision Makers Guide to Microsoft Fabric - Hedging your Fabric Bet Welcome to the first part in a new series of interviews with real-world Decision Makers (CTOs, CIOs, Heads of / Directors of Software Engineering, Data & Analytics) about how they manage their strategic roadmap and evaluate new technologies to simplify their portfolio, deliver better outcomes for stakeholders, or give them a competitive advantage. In a 3-part interview we talk to Tom Peplow about his assessment of Microsoft Fabric. TLDR; Tom Peplow, Principal & Senior Director Product Strategy at Milliman, chats with Ian Griffiths & Ed Freeman from endjin about how Microsoft Fabric is a disruptive technology. Milliman has been using Azure since 2010, building a bespoke actuarial solution which incorporates big compute and big data technologies to perform petabyte scale data processing. Milliman are currently evaluating whether Fabric can solve their problems more efficiently than their current implementation. Tom is impressed with Microsoft's internal Finance Team's digital transformation case study, which exemplifies the improvements and productivity gains his customers desire, and believes Fabric could deliver the same type of outcome. However, there are concerns that this transformation may lead to historic headaches (like Microsoft Access) all over again, causing apprehension on the IT side. Tom believes that standards are a significant enabler and that openness is key. The talk contains the following chapters: 00:00 Introduction 00:57 What does Fabric offer? 03:11 Costs and benefits of change 05:11 Fabric, Power BI and the end user experience 07:02 New enabling technology in Fabric (OneLake) 08:55 Open Table Format 10:20 Fabric's end user focus 11:53 Fear of Fabric 13:15 Guard rails for end users 13:55 DevOps & FinOps 16:16 Version Control From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Microsoft Fabric End to End Demo Series: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices Microsoft Fabric First Impressions: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric A Tour Around Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/9kd_cekeBbM 1071 2023-07-11T06:30:00Z yes no https://endjin.com/what-we-think/talks/microsoft-fabric-ingesting-5gb-into-a-bronze-lakehouse-using-data-factory https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/microsoft-fabric-ingest-5gb-into-a-bronze-lakehouse-using-data-factory.png Microsoft Fabric - Ingesting 5GB into a Bronze Lakehouse using Data Factory In this video Ed Freeman continues the Microsoft Fabric End-to-End demo series by seeing how we can quickly ingest ~5GB of data from an unauthenticated HTTP data source into OneLake using Data Factory in Microsoft Fabric. We'll see the distinction between Tables and Files in a Fabric Lakehouse, and look at how we can preview data in the Lakehouse explorer. The talk contains the following chapters: 00:00 Introduction 00:15 Dataset recap 01:25 Workspace and pipeline artifacts 01:57 Pipeline UI layout 02:21 Copy data activity options 03:07 Configure copy data activity source 05:00 Configure copy data activity destination 06:21 Add dynamic content for destination Lakehouse filepath 08:21 Copy Data activity additional settings 09:00 Manually trigger pipeline 09:21 Alternative parameterized pipeline 11:38 Reviewing pipeline run details 12:10 Default workspace artifacts 13:04 Viewing Lakehouse Files 13:46 Roundup and outro Useful links: 📖 UK Land Registry data Microsoft Fabric End to End Demo Series: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Part 3 - Testing Notebooks Part 4 - Task Flows Part 5 - Observability Microsoft Fabric First Impressions: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric A Tour Around Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/yptYlWLsVYk 867 2023-07-07T23:06:00Z yes no https://endjin.com/what-we-think/talks/microsoft-fabric-inspecting-28-million-row-dataset https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/microsoft-fabric-dataset-and-problem-statement.png Microsoft Fabric - Inspecting 28 Million row dataset In this video Ed Freeman continues the Microsoft Fabric End-to-End demo series by looking at the dataset we'll be using, and the problem statement we're trying to solve. For a data platform, we need some data! In this series we're going to be using Land Registry data provided by the UK government which registers the ownership of land and property in England and Wales. The dataset is almost 5GB in size, and provides different types of files for complete or incremental processing. This will allow us to benefit from UPSERT-like functionality enabled by #DeltaLake without having to load all the data every time we receive new information. In this video we'll take a quick look at the data, where it comes from and what format it's in, and we'll also frame up the insights we're aiming to achieve from the analysis taking place in this series. We'll finish by stepping through a sample architecture diagram - a powerful way to visualize involved data platforms at a high-level. The talk contains the following chapters: 00:00 Introduction 00:17 Sample data introduction 00:58 Sample data inspection 03:54 Insight Discovery and defining goals 06:46 Fabric architecture walkthrough 11:32 Outro Useful links: 📖 UK Land Registry data Microsoft Fabric End to End Demo Series: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Part 3 - Testing Notebooks Part 4 - Task Flows Part 5 - Observability Microsoft Fabric First Impressions: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric A Tour Around Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/miDa4FZY7GU 733 2023-06-26T23:06:00Z yes no https://endjin.com/what-we-think/talks/microsoft-fabric-lakehouse-and-medallion-architecture https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/microsoft-fabric-lakehouse-and-medallion-architecture.png Microsoft Fabric - Lakehouse and Medallion Architecture In this video Ed Freeman looks at the Lakehouse and Medallion Architecture concepts, and how they relate to Microsoft Fabric. Lakehouses have become a common foundational design pattern for modern data platforms, and have risen in popularity over the last few years after tools like Databricks, and now Microsoft Fabric, have standardized on the design. The Medallion Architecture refers to a three-tiered architectural blueprint, with data moving through different stages of state, each stage serving different purposes. The talk contains the following chapters: 00:00 Intro 00:22 What is a Lakehouse? 02:10 What is a Lakehouse in Microsoft Fabric? 04:43 Table and File structure in OneLake 05:19 Justification for Medallion architecture 06:13 Medallion architecture overview 07:00 Bronze layer 08:15 Silver layer 08:45 Gold layer 09:50 Replayability & idempotency 10:56 Alternative structures 11:52 Outro Useful links: 📖 UK Land Registry data Microsoft Fabric End to End Demo Series: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Part 3 - Testing Notebooks Part 4 - Task Flows Part 5 - Observability Microsoft Fabric First Impressions: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric A Tour Around Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/x_CvCwSbRZI 751 2023-06-20T23:06:00Z yes no https://endjin.com/what-we-think/talks/a-tour-around-microsoft-fabric https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/a-tour-around-microsoft-fabric.png A Tour Around Microsoft Fabric In this video Ed Freeman, Senior Data Engineer, who spent the last 6 months in the Microsoft Fabric Private Preview, takes you on a tour around Microsoft Fabric. Ed covers: The Fabric Portal, Fabric "Experiences", Fabric Workspace, Notebooks, SQL Endpoints, Data Factory Pipelines and much, much more! The full transcript is available below. The talk contains the following chapters: 00:00 Brief introduction to Microsoft Fabric 00:38 Fabric portal landing page and "experience" overview 02:32 Navigating through Fabric experiences 03:58 Fabric Workspace view - exploring Fabric artifact types 04:25 Quick look at Notebooks within Fabric 05:49 Brief introduction to SQL Endpoints in Fabric 06:35 Data Factory pipelines in Fabric 07:10 Azure Synapse vs Microsoft Fabric 08:45 Artifact tabs - multi-tasking experience in Fabric 09:20 Fabric Workspace UI - filtering and changing views 10:00 Additional workspace features 10:42 Workspace roles 11:10 Monitoring Hub in Microsoft Fabric 11:58 Wrap-up From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Part 3 - Testing Notebooks Part 4 - Task Flows Part 5 - Observability Microsoft Fabric End to End Demo: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices If you want to learn more about Fabric, take a look at some of our other content: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/Qu6my9b2FWg 757 2023-05-23T23:06:00Z yes no https://endjin.com/what-we-think/talks/perspectives-on-microsoft-fabric https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/perspectives-on-microsoft-fabric.png Perspectives on Microsoft Fabric Microsoft Fabric is a third generation data and analytics platform, building on strong foundations that have been established by Azure Synapse Analytics and Power BI. It is a SaaS solution that further lowers barriers to adoption. Endjin has been on the Private Preview for Microsoft Fabric since late 2022, and have put the platform through its paces using real-world data and scenarios. In this 20 minute chat, Microsoft MVP Ian Griffiths interviews Barry Smart, Director of Data & AI, and Ed Freeman, Senior Data Engineer, about their experiences of this new unified data platform. The full transcript is available below. The talk contains the following chapters: 00:50 What is Microsoft Fabric? 02:12 How does that compare with other offerings like Databricks or Snowflake? 03:43 What does the transition from PaaS to SaaS mean? 05:41 Does Fabric represent a shift in maturity and how would that look on a Wardley Map? 07:41 What have you been doing with Fabric? 08:30 Do you have a favourite feature? 09:40 Is Fabric ready for prime time? 12:19 Could you sketch out how you might approach building a spike to explore Fabric? 15:20 Ian's final thoughts 16:02 Barry's final thoughts 17:23 Ed's final thoughts From Descriptive to Predictive Analytics with Microsoft Fabric: Part 1 - Overview Part 2 - Data Validation with Great Expectations Part 3 - Testing Notebooks Part 4 - Task Flows Part 5 - Observability Microsoft Fabric End to End Demo Series: Part 1 - Lakehouse & Medallion Architecture Part 2 - Plan and Architect a Data Project Part 3 - Ingest Data Part 4 - Creating a shortcut to ADLS Gen2 in Fabric Part 5 - Local OneLake Tools Part 6 - Role of the Silver Layer in the Medallion Architecture Part 7 - Processing Bronze to Silver using Fabric Notebooks Part 8 - Good Notebook Development Practices Microsoft Fabric First Impressions: Perspectives on Microsoft Fabric Introduction to Microsoft Fabric A Tour Around Microsoft Fabric Decision Maker's Guide to Microsoft Fabric Hedging your Fabric Bet Microsoft Fabric and Generative AI The Pace of Innovation and find all the rest of our content here. https://www.youtube.com/embed/t3K4YEtvvLs 1149 2023-05-23T23:05:00Z yes no https://endjin.com/what-we-think/talks/creating-a-high-performance-data-team-lessons-learned-from-the-field https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/sqlbits-2023-how-to-create-a-high-performance-dat-team.png Creating a high performance Data Team: lessons learned from the field There are many factors that influence the ability for data and analytics teams to achieve their full potential. With most organisations seeking to become data driven, how can they build a data and analytics team that is able to meet increasing demands in an agile, cost effective and sustainable way? In this session, Barry Smart will explore the factors that lead to success (and the pitfalls to avoid). He will use real world examples to bring it to life based on his experience delivering data intensive, high business impact projects over the last 25 years. https://www.youtube.com/embed/xB3_g_rGvPQ 998 2023-04-06T00:00:00Z yes no https://endjin.com/what-we-think/talks/building-large-scale-multi-tenant-saas-systems-with-azure-cosmos-db https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/cosmos-db-conf-2023-building-large-scale-multi-tenant-saas-systems.jpg Building Large-Scale Multi-Tenant SaaS Systems with Azure Cosmos DB SaaS platforms with many tenants need to be able to scale, and also to maintain isolation between customers, while keeping costs under control. This session explains how we exploited Azure Cosmos DB's effectively limitless scale and security features to provide high-performance, cost-effective storage while maintaining strict security and performance isolation between tenants. https://www.youtube.com/embed/jD3KNfW2ar8 825 2023-03-29T00:00:00Z yes no https://endjin.com/what-we-think/talks/an-overview-of-reaqtor-aka-cloud-native-rx https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/talk-dnf-summit-2023-reaqtor-reliable-rx-at-scale.png An overview of Reaqtor AKA Cloud Native Rx Endjin are proud to be a .NET Foundation Corporate Sponsor, as we are maintainers of Reactive Extensions for .NET, Reaqtor (both of which are part of the .NET Foundation), and over 50 of our own Open Source Projects. In this talk, as part of the .NET Foundation Summit 2023, Ian Griffiths gives a high level overview of Reaqtor and explains how it evolves many of the core concepts in Reactive Extensions for .NET (Rx), to support Cloud Native Reactive Programming. https://www.youtube.com/embed/SCi6mghCRmc 3212 2023-01-17T00:00:00Z yes no https://endjin.com/what-we-think/talks/azure-synapse-database-templates-in-10-minutes https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/talk-poster-azure-synapse-database-templates-in-10-minutes.png Azure Synapse Database Templates in 10 Minutes Database templates are a recent new feature added to Azure Synapse Analytics. They allow you to put a layer of abstraction over your data lake to make it appear more like a relational system than an file system. This is a an example of how Microsoft continue to invest in Synapse Analytics, adding yet another tool to the Synapse "Swiss Army Knife". In this short talk, Barry provides an overview of Database Templates and illustrates how they can be put to use within a modern, cloud native, data & analytics architecture. He uses a practical example to show how Database Templates can be leveraged by putting them at the heart of the process to design, build, test and deploy actionable insights on Azure Synapse Analytics. https://www.youtube.com/embed/Us1mgW2QwZY 722 2022-12-06T00:00:00Z yes no https://endjin.com/what-we-think/talks/accessible-data-storytelling-with-power-bi-design-concepts-and-accessible-colours https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/power-bi-accessible-design-and-usability-for-custom-reports.png Accessible Data Storytelling with Power BI: Design Concepts and Accessible Colours We've created a report that uses open data published by the World Bank to explore global trends in wealth & health. This report has been published to the Power BI community data stories gallery. The video below walks through the report from a visual design perspective. Red–green color blindness affects up to 1 in 12 males (8%) and 1 in 200 females (0.5%)1 2. It reinforces how important concepts such as branding, accessibility, visual design, colour palette choice. interactivity and page layout are to allowing users to engage successfully with a report. You can find the report on the Power BI Community Data Storytelling Gallery. You can also watch a high-level walk-through of the report: Data Storytelling with Power BI: The World Bank World Health and Wealth Report "Facts About Color Blindness". NEI. February 2015. Archived from the original on 28 July 2016.↩ "Colour vision deficiency (colour blindness)". nhs.uk. 18 October 2017.↩ https://www.youtube.com/embed/lVUVqlUKOhU 489 2022-11-30T06:30:00Z yes no https://endjin.com/what-we-think/talks/data-storytelling-with-power-bi-the-world-bank-world-health-and-wealth-report https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/power-bi-data-storytelling-world-bank-world-health-and-wealth-report.png Data Storytelling with Power BI: The World Bank World Health and Wealth Report We've created a report that uses open data published by the World Bank to explore global trends in wealth & health. This report has been published to the Power BI community data stories gallery. The video below walks through the report. It is an example of how technology, data and visual design can be combined to engage people and turn data into actionable insights. The report captures many of the techniques we use when we develop analytics for our clients. We find that design concepts such as branding, accessibility and persona driven user journeys are key to enabling the report to have the impact intended. You can find the report on the Power BI Community Data Storytelling Gallery. You can also watch a discussion about brand, design, usability and accessibility of the report: Accessible Data Storytelling with Power BI: Design Concepts and Accessible Colours https://www.youtube.com/embed/JCauiWwPXlk 418 2022-11-30T06:30:00Z yes no https://endjin.com/what-we-think/talks/do-those-numbers-look-right https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/do-those-numbers-look-right.png Do Those Numbers Look Right? How to ensure quality, and avoid inaccuracies in your data insights. The need to validate business rules and security boundaries within a data solution is important, as well as the need for ensuring that quality doesn't regress over time as it evolves. Data insights are useless or even dangerous if they can't be trusted, and in this session, James will explain how and why they should treated just like any other software project with respect to testing - by building automated quality gates into the end to end development process. During the session, he'll walkthrough some practical examples and proven techniques around testing data solutions - including Power BI reports, Synapse Pipelines and interactive Spark notebooks. https://www.youtube.com/embed/YDFsVLODCtM 3409 2022-10-26T00:00:00Z yes no https://endjin.com/what-we-think/talks/fake-it-til-you-make-it-generating-production-quality-test-data-at-scale https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/generating-production-quality-test-data-at-scale.png Fake it 'til you make it - generating production quality test data at scale. Many organisations provide digital products or services that need to handle personally identifiable information. The challenge is providing product and engineering teams with a sufficient volume of realistic looking synthetic data to enable them to design, develop and test their solutions. Barry presents open source tools and open data sources that can be used to tackle this challenge, and then demos this in action to generate thousands of synthetic customers. He describes how this approach can be used to build better products, to test products using production quality data at production scale, and embed data quality and best practice information security practices in your engineering processes. https://www.youtube.com/embed/pcc-EXzK3vI 2985 2022-09-13T06:30:00Z yes no https://endjin.com/what-we-think/talks/how-to-dynamically-create-sql-serverless-views-from-azure-synapse-pipelines https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/blog/2022/02/data-toboggan-how-to-dynamically-create-sql-serverless-views.png How to dynamically create SQL Serverless views from Azure Synapse Pipelines SQL Serverless allows you to add structured schemas over standard file formats in your data lake storage. For all the benefits this brings, one of the trade offs is that your schema is only enforced "on read", rather than "on write". So if it's your Synapse Pipelines that are responsible for writing/updating the data that is stored in your data lake, then they should also be responsible for updating the schema of the SQL Serverless views that you use to query it. In this session, James Broome will show a simple way to dynamically update SQL Serverless views from with a Synapse Pipeline so that they're always up to date with the data in your data lake. https://www.youtube.com/embed/jE_-QLN7GT0 736 2022-02-04T00:00:00Z yes no https://endjin.com/what-we-think/talks/how-to-test-azure-synapse-notebooks https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/blog/2022/02/data-toboggan-how-to-test-azure-synapse-notebooks.png How to test Azure Synapse notebooks Interactive Notebooks in Synapse are an incredibly powerful tool for data exploration and experimentation. But with this agility comes a risk - how do you ensure quality, and avoid inaccuracies in your data insights? In this session, James will show that with a simple shift in mindset around how notebooks are structured, tests can be added relatively easily to any notebook logic, just like any other form of application logic. He'll quickly walk through an end-to-end sample notebook, that can be used as a reference for your own projects. https://www.youtube.com/embed/UKMyB47ivuk 914 2022-02-01T00:00:00Z yes no https://endjin.com/what-we-think/talks/how-to-define-business-requirements-for-a-successful-cloud-data-and-analytics-project https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/how-to-define-requirements-for-a-successful-data-project.png How to define business requirements for a successful cloud data & analytics project Many data projects fail to deliver the impact they should for a simple reason – they focus on the data. In this session, James explains a different way of thinking that will set up your data & analytics projects for success. Using an iterative, action-oriented, insight discovery process, he will demonstrate tools and techniques that will help you to identify, define and prioritize requirements in your own projects so that they deliver maximum value. He'll also explore the synergy with modern cloud analytics platforms like Azure Synapse, explaining how the process and the architecture actively support each other for fast, impactful delivery. https://www.youtube.com/embed/X1W7_Xm1o8w 1365 2022-01-27T00:00:00Z yes no https://endjin.com/what-we-think/talks/monetize-apis-with-azure-api-management-using-adyen https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/monetize-apis-with-azure-api-management-using-adyen.png Monetize APIs with Azure API Management Using Adyen If you've already watched our tech briefing unlocking digital transformation using Azure API Management and you've run our API Maturity assessment for your organization, your next big problem to solve is how to monetize your APIs. In this tutorial, we walk you through this demo project, where the example revenue model defined in the monetization overview (Free, Developer, PAYG, Basic, Standard, Pro, Enterprise) has been implemented to demonstrate integrating Azure API Management with Adyen. This demo enables API consumers to browse, select, and subscribe to products via the developer portal. We have provided detailed technical implementation about the overall architecture and payment flow, as well as a step-by-step deployment guide, including the Adyen developer portal. https://www.youtube.com/embed/eVbgryEub6M 464 2022-01-26T00:00:00Z yes no https://endjin.com/what-we-think/talks/monetize-apis-with-azure-api-management-using-stripe https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/monetize-apis-with-azure-api-management-using-stripe.png Monetize APIs with Azure API Management Using Stripe Once you've understood how to unlock digital transformation using APIs and assessed your organization's API Maturity, the next big problem to solve is how to monetize your APIs. In this tutorial, we walk you through this demo project, where the example revenue model defined in the monetization overview (Free, Developer, PAYG, Basic, Standard, Pro, Enterprise) has been implemented to demonstrate integrating Azure API Management with Stripe. This demo enables API consumers to browse, select, and subscribe to products via the developer portal. We have provided detailed technical implementation about the overall architecture and process flow, as well as a step-by-step deployment guide, including the Stripe developer portal. https://www.youtube.com/embed/tO4UMGbrgrQ 717 2022-01-26T00:00:00Z yes no https://endjin.com/what-we-think/talks/reaqtor-reliable-rx-at-scale-for-high-performance-event-processing https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/dotnet-oxford-reaqtor-reliable-rx-at-scale-for-high-performance-event-processing.jpg Reaqtor - Reliable Rx at Scale for High-performance Event Processing The Reactive Extensions for .NET (Rx) is one of the most influential technologies to have emerged from the .NET ecosystem. (Its influence on other platforms has been so pervasive that its origins in .NET are sometimes forgotten.) The team that created Rx did not rest on their laurels: they used Rx as the basis for developing a distributed, reliable, and extremely scalable event processing service. Reactor, as it was then called, has been integral to some of Microsoft's most widely used services for many years, including Cortana and Office 365, but although Microsoft has talked in public about it a few times, it has never been available for use outside of Microsoft—until now. On May 18th 2021, Bart de Smet, made Reaqtor (its new name) open source, as a .NET-Foundation-sponsored project. This talk will explain what makes Reaqtor different from other high-scale event processing systems. It will also show some of the foundational components that have been released as part of this, most notably 'Bonsai', a mechanism for serializing computations based on .NET's expression tree system, and which is central to how Reaqtor works. https://www.youtube.com/embed/K5A3uP75XNQ 4431 2021-06-22T00:00:00Z yes no https://endjin.com/what-we-think/talks/how-to-start-your-career-in-data-science https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/how-to-start-your-career-in-data-science-2021.png How to Start Your Career in Data Science In this meet up, Barry Smart shares his career journey: from software engineer, to architect, to IT Director, to CTO, and then going back to University to get a Masters in Artificial Intelligence and Applications. https://www.youtube.com/embed/vJrBkxeaOFs 4990 2021-06-09T00:00:00Z yes no https://endjin.com/what-we-think/talks/how-to-safely-reference-a-nullable-activity-output-in-azure-synapse-pipelines-and-azure-data-factory https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/how-to-safely-reference-a-nullable-activity-output-in-azure-data-factory.png How to safely reference a nullable activity output in Azure Synapse Pipelines and Azure Data Factory Did you know that there's a null-safe operator in Azure Data Factory's expression syntax? No? Well, now you do. Watch the video to see how you can safely reference an activity output that might not always exist. https://www.youtube.com/embed/V4m46NfEp60 360 2021-02-02T00:00:00Z yes no https://endjin.com/what-we-think/talks/beyond-modern-data-warehouse-mvp-panel-discussion https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/beyond-modern-data-warehouse-panel-microsoft-reactor.jpg Beyond Modern Data Warehouse MVP Panel Discussion New big data technologies bring new possibilities. Azure Synapse Analytics, Azure Purview, Azure Databricks, Spark, Power BI, DAX, and Machine Learning are transforming how organisation approach building and governing modern data platforms to unlock the potential of data, information and actionable insights. Does this change mean that all previous skills, knowledge, IP, and experience has to be thrown away, or can existing investments be reused? What does the career pathway look like for DBAs, data engineers, Business Intelligence specialists? Does everyone now have to learn Python, Scala, Spark, Parquet and Notebooks, or is there some longevity with exiting SQL, PowerShell and C# skills? Can business analysts use these tools to reduce time to insights and break free of the limitation of Excel and SQL Server Management Studio to obtain answers to business questions? Will these modern data platforms create entirely new job roles? Do these tools help break down the silos across organisations to allow different personas to collaborate on data and insights in a way they haven't been able to before? And most importantly, will these advancements enable to better outcomes for the end users who don't really care about details, but just want information they can trust in order to drive decision making? These topics and more are discussed by a panel of Microsoft MVPs including endjin co-founder Howard van Rooijen. https://www.youtube.com/embed/LEx-KIUR08U 3784 2020-12-14T00:00:00Z yes no https://endjin.com/what-we-think/talks/wardley-maps-can-maps-do-good https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/mapcamp-2020-can-maps-do-good.png Wardley Maps: Can Maps do Good? Can maps do good? We discuss whether maps are an effective tool in the effort to make change for good. We look at approaches to determining what good looks like, how maps can help us understand where we really are, to find paths to better outcomes, and how they can be used in combination with other tools and data to do better. https://www.youtube.com/embed/CE5OJRc0Gh0 1800 2020-11-13T00:00:00Z yes no https://endjin.com/what-we-think/talks/building-a-secure-data-solution-using-azure-data-lake-store https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/sqlbits-2020-building-a-secure-azure-data-lake-storage-architecture.png Building a secure data solution using Azure Data Lake Store There are many different cloud storage solutions, each with different applications and benefits. In this talk we will focus on the benefits of using Azure Data Lake Storage (ADLS). We will cover how we use ADLS in real projects to produce a security architecture which is adaptable, scalable and provides in-built monitoring capabilities. We will walk through in more detail how ADLS can be combined with serverless processing and secured via Azure Active Directory in order to gain insights without exposing more data than necessary. By the end of this talk you will have a good understanding of the advantages of using ADLS, and the different techniques and patterns we regularly use to build up secure and manageable data architectures. https://www.youtube.com/embed/9cdfCl2LZ2g 2274 2020-11-02T00:00:00Z yes no https://endjin.com/what-we-think/talks/navigating-the-bewildering-array-of-data-services-in-azure https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/sqlbits-2020-navigating-the-bewildering-array-of-data-services-in-azure.png Navigating the Bewildering Array of Data Services in Azure Azure offers multiple options for each aspect of data handling. It offers multiple relational databases, but there are also completely different models such as table and blob storage, data lakes, or the multi-model approach taken by CosmosDB. Even if you decide on SQL Server, there are several different deployment and pricing models. For some applications, one of the analytics solutions might be a good fit, but how do you decide between Azure Synapse Analytics, Azure Stream Analytics, or Azure Analysis Services. Is Azure Data Factory a good fit for your ETL needs, and if so, will a pipeline built of standard activities meet your needs or will you need to create custom data flows? This talk will help you to navigate the many options, so that you can quickly narrow down a small set of candidate technologies that you can then assess to make the right decisions. https://www.youtube.com/embed/ziLlfT9c2RY 2943 2020-10-23T00:00:00Z yes no https://endjin.com/what-we-think/talks/data-exploration-and-experimentation-with-notebooks-in-azure https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/sqlbits-2020-data-exploration-and-experimentation-with-notebooks-in-azure.png Data Exploration & Experimentation with Notebooks in Azure Microsoft are investing heavily in Notebook technologies (such as Jupyter and .NET Interactive) to provide an interactive environment for experimental and exploratory data inspection and analysis. These kinds of environment are becoming increasingly important for a growing range of activities including data cleaning and normalization, data import, statistical analysis, insight generation, and testing hypotheses. And in some applications, it can make sense to take notebooks that started out as an interactively developed set of ad hoc operations and transform them into part of an automated workflow. Jupyter notebooks are most commonly authored in Python or R. Microsoft has been working to add support for .NET languages, enabling the use of C# and F# in notebooks. They are also adding .NET support for Spark, enabling Spark clusters to be controlled from .NET, and also with a view to being able to run custom .NET code inside the cluster as part of the core processing. Azure's growing support for notebooks enables this approach across a range of scales. You can work with datasets that fit easily in a single machine's memory, but if you need more firepower, with Azure's Databricks support you can spin you up a server farm to process your data in parallel, enabling you to perform complex computations across massive datasets. https://www.youtube.com/embed/SFfpR0I-K3Q 2792 2020-10-16T00:00:00Z yes no https://endjin.com/what-we-think/talks/detecting-anomalies-in-iot-telemetry-with-azure-synapse-analytics https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/detecting-anomalies-in-iot-telemetry-with-azure-synapse-analytics-sqlbits2020.png Detecting Anomalies in IoT Telemetry with Azure Synapse Analytics This talk explores a real-world predictive maintenance use case, focusing on anomaly detection in device telemetry (ioT) to aid customer service. As well as looking at the technology behind Azure Synapse Analytics, it will explain how it supports the end to end data science process, providing a reference point for getting started with other advanced analytical workloads. https://www.youtube.com/embed/TswRY3uprtE 2440 2020-10-09T00:00:00Z yes no https://endjin.com/what-we-think/talks/what-makes-a-good-api https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/api-specification-conference-2020-talk-what-makes-a-good-api.png What makes a good API? When you delve into GraphQL, oDATA, Hydra and pals, you see a lot of very clever, highly generalized, very "self-describing" API design patterns. But sometimes we lose sight of the fact that, above all, we want people to be able to consume our APIs, to derive business value. Quickly. Effectively. With a minimum of fuss. In this session we are going to talk about APIs from the consumer's point of view. What is "easy" to understand for the average developer? Where do common patterns create obstacles? Or bugs? How can we keep it simple? We're going to look at some popular APIs and analyse them for ease of understanding, and look at the kind of code that's generated by popular client tooling, to see which patterns work well, and which get in the way. At the end of the session we'll walk away with some API design patterns that are less "clever", but much simpler. And therefore more beautiful. https://www.youtube.com/embed/mXFoZBBw7JI 339 2020-09-25T00:00:00Z yes no https://endjin.com/what-we-think/talks/reactive-extensions-for-dotnet https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/reactive-extensions-for-dotnet-dotnetsheff.png Reactive Extensions for .NET The Reactive Extensions for .NET ("Rx" for short) are a set of libraries for working with asynchronous and event-based information sources. While Rx has become widely adopted in client-side code, its roots are in highly scalable server-side processing. In this talk, Ian Griffiths shows why Rx should be a critical part of any .NET developer's toolbox, and shows a real-world application that illustrates how the powerful processing and orchestration facilities Rx provides can be applied to collect and process data in edge devices and then stream the high-value events to an Azure IoT hub for further analysis in the cloud. https://www.youtube.com/embed/6yjl_h7-WYA 4431 2020-08-06T00:00:00Z yes no https://endjin.com/what-we-think/talks/serverless-data-prep-using-sql-on-demand-and-synapse-pipelines https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/serverless-data-prep-using-sql-on-demand-and-synapse-pipelines.png Serverless data prep using SQL on demand and Synapse Pipelines Azure Synapse Pipelines (Azure Data Factory) can run SQL on demand jobs to achieve serverless data preparation for your data pipelines. This video walks through the process of building a serverless data prep pipeline. It shows how to take advantage of the TDS endpoint of SQL on demand to leverage the built-in Synapse Pipelines copy activity. It also points out a couple of existing gotchas that may catch you out along the way. https://www.youtube.com/embed/NlYNcAduQxE 457 2020-07-08T00:00:00Z yes no https://endjin.com/what-we-think/talks/highlights-from-sustainable-finance-live-a-gathering-of-real-world-expertise https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/finextra-tv-sustainable-finance-live-2020-highlights.jpg Highlights from Sustainable Finance.LIVE: A Gathering of Real-World Expertise Speakers from Sustainable Finance.LIVE gather post-event to talk about why they took part in the Co-Creation Workshops, their key takeaways, and how they went about solving the challenges the community is finding in the Investment and Asset Management space. Matthew Adams & Jess Panni from endjin, lead Sustainable Finance.Live's session on Reporting and Data to Build an ESG Portfolio and guided expert attendees through working in Data Management while discussing the challenges of building a Passive investing Index drive portfolio for Institutional Investors using ESG and other data. They use Wardley Mapping as a collaboration tool to explore the problem space. https://www.youtube.com/embed/aRHAULTvaks 339 2020-06-23T00:00:00Z yes no https://endjin.com/what-we-think/talks/azure-synapse-on-demand-serverless-compute-and-querying https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/microsoft-mechanics-azure-synapse-on-demand-serverless-compute-and-querying.jpg Azure Synapse - On-Demand Serverless Compute and Querying Find out how UK-based data engineering consultancy, endjin, is evaluating Azure Synapse for on-demand serverless compute and querying. Endjin specializes in big data analytics solutions for customers across a range of different industries such as ocean research, financial services, and retail industries. Host Jeremy Chapman speaks with Jess Panni, Principal and Data Architect at endjin, to discuss how they're using SQL serverless for on-demand compute as well as visualization capabilities to help customers with big data challenges. https://www.youtube.com/embed/JXdFAEYCO-M 754 2020-06-17T00:00:00Z yes no https://endjin.com/what-we-think/talks/custom-csharp-spark-jobs-in-azure-synapse https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/azure-synapse-custom-csharp-jobs.jpg Custom C# Spark Jobs in Azure Synapse Azure Synapse workspaces can host a Spark cluster. In addition to providing the execution environment for certain Synapse features such as Notebooks, you can also write custom code that runs as a job inside Synapse hosted Spark cluster. This video walks through the process of running a C# custom Spark job in Azure Synapse. It shows how to create the Synapse workspace in the Azure portal, how to add a Spark pool, and how to configure a suitable storage account. It also shows how to write the custom job in C#, using Visual Studio Code's support for Docker-based development to target the exact JDK and Scala version used in Synapse. You will see how to upload the built output to Azure, and then how to configure Azure Synapse to execute the resulting JAR as a custom job. https://www.youtube.com/embed/N2QB8XXR6PE 919 2020-06-16T00:00:00Z yes no https://endjin.com/what-we-think/talks/custom-scala-spark-jobs-in-azure-synapse https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/azure-synapse-custom-scala-jobs.jpg Custom Scala Spark Jobs in Azure Synapse Azure Synapse workspaces can host a Spark cluster. In addition to providing the execution environment for certain Synapse features such as Notebooks, you can also write custom code that runs as a job inside Synapse hosted Spark cluster. This video walks through the process of running a Scala custom Spark job in Azure Synapse. It shows how to create the Synapse workspace in the Azure portal, how to add a Spark pool, and how to configure a suitable storage account. It also shows how to write the custom job in Scala, using Visual Studio Code's support for Docker-based development to target the exact JDK and Scala version used in Synapse. You will see how to upload the built output to Azure, and then how to configure Azure Synapse to execute the resulting JAR as a custom job. https://www.youtube.com/embed/wDzigDa6CHk 790 2020-06-16T00:00:00Z yes no https://endjin.com/what-we-think/talks/uniting-the-world-of-data-with-esg-goals https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/finextra-tv-uniting-the-world-of-data-with-esg-goals.jpg Uniting the World of Data with ESG Goals Matthew Adams from endjin talks about the good of the SustainableFinance.Live space and the positive outcomes from the event, how the world of data science and corporate ESG can be united to achieve our sustainable goals. https://www.youtube.com/embed/Lpx2RUHqz9I 339 2020-06-10T00:00:00Z yes no https://endjin.com/what-we-think/talks/a-view-from-reporting-and-data-to-build-an-esg-portfolio https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/finextra-tv-reporting-and-data-to-build-an-esg-portfolio.jpg A View From: Reporting and Data to Build an ESG Portfolio Jess Panni, Principal at endjin lead SustainableFinance.Live's session on Reporting and Data to Build an ESG Portfolio and guided expert attendees through working in Data Management while discussing the challenges of building a Passive investing Index drive portfolio for Institutional Investors using ESG and other data. He talks about overcoming these challenges and identifying opportunities to create sound portfolios that deliver returns for all stakeholders. https://www.youtube.com/embed/gdX9lZpPnUA 434 2020-06-09T00:00:00Z yes no https://endjin.com/what-we-think/talks/5-ways-to-prepare-for-public-speaking https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/5-ways-to-prepare-for-public-speaking.png 5 ways to prepare for public speaking Public speaking can be a nerve-wracking prospect. Having spoken at NDC in January, and Azure Oxford in February, Carmel Eve runs through a few tips on how to prepare for public speaking! https://www.youtube.com/embed/8UvEv-WjBR8 319 2020-03-30T00:00:00Z yes no https://endjin.com/what-we-think/talks/cnab-packages-to-deploy-applications-from-azure-to-anywhere-at-all https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/cnab-packages-to-deploy-applications-from-azure-to-anywhere-at-all.jpg CNAB packages to deploy applications from Azure to anywhere at all CNAB (Cloud-Native Application Bundle) is a new specification designed for facilitating the bundling, installing and managing of cloud-native or distributed apps. It provides you with a way to containerise your installation process, and easily deploy & share your entire solution across any cloud, on-prem, or at the edge. In this session Mike Larah provides a high-level overview of CNAB and the Azure CNAB Quickstart Library and pairs with Michael Crump to create a simple CNAB Hello World example. https://www.youtube.com/embed/2msE9eB2FM0 6354 2020-03-30T00:00:00Z yes no https://endjin.com/what-we-think/talks/paths-into-technology https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/paths-into-technology.png Paths into technology There are many different paths into the technology industry. In this video Carmel Eve discusses how she entered the industry from a science background, and how science equips you for a career in technology. Through this, she hopes to promote the idea that you don't need to specialise early in order to succeed in tech! https://www.youtube.com/embed/1457cRimaVg 154 2020-03-22T00:00:00Z yes no https://endjin.com/what-we-think/talks/managing-mental-health-and-remote-working https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/managing-mental-health-and-remote-working.png Managing mental health and remote working Endjin have been now been a fully remote company since 2017. Remote working provides a lot of benefits in terms of work-life balance and freedom. However, it can also be challenging for your mental health. With many more people now working remotely worldwide, Carmel shares some of the ways that she manages these challenges. https://www.youtube.com/embed/p0Tng6LU4uI 458 2020-03-12T00:00:00Z yes no https://endjin.com/what-we-think/talks/csharp-8-0-nullable-reference-in-practice https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/dotnet-oxford-csharp-8-nullable-reference-in-practice.jpg C# 8.0 Nullable Reference in Practice The most significant new feature of C# 8.0 is its support for non-nullable references. In a bid to mitigate what is often described as the "billion dollar mistake" of allowing references to be null, C#'s language designers have attempted to retrofit the ability to write code that declares unambiguously that it doesn't want nulls, and for the compiler to attempt to enforce that for you. Adding such a feature almost two decades into the life of the language and its supporting runtime was always going to be challenging, so this feature has quite a few quirks (such as its apparently inverted name). In this talk, Ian Griffiths will describes the ins and outs of the new feature, with particular reference to his employer, endjin's experience with applying this new feature to existing code. In addition to an in-depth description of the technical details of nullable references, this talk will also describe strategies for migrating an existing codebase into the world of nullability-awareness. https://www.youtube.com/embed/0IH_oLe9Bn0 4954 2020-03-10T00:00:00Z yes no https://endjin.com/what-we-think/talks/introduction-to-azure-cnab-quickstarts https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/introduction-to-azure-cnab-quickstarts.png Introduction to Azure CNAB Quickstarts Over the last few months, endjin have been working together with Microsoft to create the Azure CNAB Quickstart Library in GitHub. CNAB (Cloud-Native Application Bundle) is a new specification designed for facilitating the bundling, installing and managing of cloud-native or distributed apps. It provides you with a way to containerise your installation process, and easily deploy & share your entire solution across any cloud, on-prem, or at the edge. We're excited about CNAB as it offers a much needed abstraction for defining all the required artefacts and steps to install your applications, bringing together whatever multitude of tools, their pre-requisites, and the runbooks required. We created the Quickstarts Library so that the community can discover and use bundles for deploying applications and solutions, learn how to create their own bundles, and contribute back to the library. The library is designed to be optimized for bundles that use Azure resources, but is not limited to Azure only. https://www.youtube.com/embed/z1lnQfaAVeg 692 2020-03-02T00:00:00Z yes no https://endjin.com/what-we-think/talks/machine-learning-and-azure-against-illegal-fishing https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/azure-oxford-talk-machine-learning-and-azure-against-illegal-fishing.jpg Machine Learning and Azure Against Illegal Fishing Illegal fishing is a worldwide problem which threatens developing nations, damages marine wildlife and supports other criminal activities like human trafficking and slavery. Recently we worked with a not-for-profit who, through machine learning and complex geo-spatial models, detects and helps to tackle this vital global issue. Through Microsoft Azure, we enabled them to more reliably, and in near-real-time, produce fishing alerts from all over the world. In this talk we will walk through how to construct a cloud-first architecture based on serverless and data analytics technologies. We will then explore the important principles and challenges in designing this kind of solution. Finally, we will see how the architecture we designed through this process not only provides all the benefits of the cloud (reliability, scalability, security), but because of the pay-as-you-go compute model, has a compute cost of less than £10 per month! https://www.youtube.com/embed/qeh_-mxnq-g 3471 2020-02-24T00:00:00Z yes no https://endjin.com/what-we-think/talks/the-art-of-wardley-mapping-sustainable-business-goals https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/the-art-of-wardley-mapping-sustainable-business-goals.jpg The art of Wardley Mapping Sustainable business goals Mapping Sustainable business goals to a technology strategy is as much art as it is computer science. Matthew Adams Founder at endjin the brains behind the Ocean Mind story, sat down with Richard Peers to discuss how they help organisations do just that using Wardley Mapping. The two then dive into the detail of the role of data which is so crucial to Sustainable Finance. https://www.youtube.com/embed/5veixfBI6uM 710 2020-02-20T00:00:00Z yes no https://endjin.com/what-we-think/talks/combatting-illegal-fishing-with-machine-learning-and-azure https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/ndc-combatting-illegal-fishing-with-machine-learning-and-azure.jpg Combatting illegal fishing with Machine Learning and Azure Illegal fishing is a worldwide problem which threatens developing nations, damages marine wildlife and supports other criminal activities like human trafficking and slavery. Recently we worked with a not-for-profit who, through machine learning and complex geo-spatial models, detects and helps to tackle this vital global issue. Through Microsoft Azure, we enabled them to more reliably, and in near-real-time, produce fishing alerts from all over the world. In this talk we will walk through how to construct a cloud-first architecture based on serverless and data analytics technologies. We will then explore the important principles and challenges in designing this kind of solution. Finally, we will see how the architecture we designed through this process not only provides all the benefits of the cloud (reliability, scalability, security), but because of the pay-as-you-go compute model, has a compute cost of less than £10 per month! https://www.youtube.com/embed/gJV30GTQE7I 2768 2020-01-29T00:00:00Z yes no https://endjin.com/what-we-think/talks/a-step-by-step-customer-propensity-experiment-with-azure-machine-learning-studio https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/customer-propensity-experiment-with-machine-learning-studio.png A step-by-step customer propensity experiment with Azure Machine Learning Studio In this video, endjineer Ed Freeman performs a "worked example" customer propensity experiment using Azure Machine Learning Studio. "Our hypothetical situation is based on an e-commerce business that is interested in changing its business processes to prioritise customers that are more likely to purchase their new service, based on the information they provide at various stages throughout the purchasing workflow." The worked example covers: 00:20 What is "Customer Propensity"? 00:35 Classification Problems 01:02 Purpose of Walkthrough 01:41 General Method 02:45 Our example 03:05 Hypothesis 03:32 Interpretation of results 04:02 Desired Metric - AUC 04:26 Worked Example in Azure Machine Learning Studio 36:35 Conclusion https://www.youtube.com/embed/_AbfMNhY91A 2242 2020-01-27T00:00:00Z yes no https://endjin.com/what-we-think/talks/wardley-maps-explaining-how-oceanmind-use-microsoft-azure-and-ai-to-combat-illegal-fishing https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/wardley-maps-explaining-how-oceanmind-use-microsoft-azure-and-ai.jpg Wardley Maps - Explaining how OceanMind use Microsoft Azure & AI to combat Illegal Fishing Illegal fishing is having a devastating impact on the bio-diversity of the worlds oceans. With 1/3 of all fish stocks under threat the challenge is enormous. OceanMind apply AI and machine learning over billions of data points to detect illegal fishing activity and report this for law enforcement and compliance. With big ambitions and ageing technology OceanMind needed a digital transformation strategy that would allow them to scale their mission without costing the earth. Digital transformation is hard enough as it is. The ever shifting sands of business and technology makes makes strategy feel like a black art, where success stories feel more like luck than judgement. This was exactly the problem that OceanMind had when faced with migrating their solution to Microsoft Azure. At endjin, we are constantly on the look out for tools that help us and our customers make better decisions, based on information, context and awareness rather than blind faith. One of our favourites is Wardley mapping, which applies the notion of maps, as a way to plan and guide you to your intended destination very literally indeed. Simon Wardley, in his eBook, points out that most business mapping techniques we are sold are not maps at all. Over the years, seemingly as a result of being bombarded by business process hype, the term map has been watered down to the point that it has very little meaning at all. This video explains how Wardley Mapping was applied to identify and justify OceanMind's successful Azure migration strategy. If you have never come across Wardley mapping before you will be able to follow along, but I strongly recommend reading Simon's eBook if you are keen to try it for yourselves. https://www.youtube.com/embed/EQfQoc8Ybdc 689 2020-01-14T00:00:00Z yes no https://endjin.com/what-we-think/talks/unlocking-digital-transformation-with-azure-api-management https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/digital-transformation-with-azure-api-management.png Unlocking Digital Transformation with Azure API Management In this 50 minute Tech Briefing, James Broome, endjin's Director of Engineering explains our perspective on digital transformation, and how this can be enabled by using APIs to unlock data and algorithms within your organisation. Digital Transformation is the buzzword of the moment; organisations are trying to reconfigure themselves to operate in the digital age. One of the key enablement strategies involves putting APIs at the heart of the organisation to centralise access to data, logic and value, while removing those traditional barriers that prevent collaboration between organisational silos. APIs are being sold as the silver bullet solution to Digital Transformation, but like any other software or product development programme, the real complexities & overall effort exist outside of the realm of coding. Embarking on the journey of designing, building, testing, deploying, supporting, marketing, metering & billing APIs can be a daunting prospect. We recommend that organisations adopt an “assess & invest” feedback loop; measure your maturity across the disciplines required to build, support & sell APIs and invest in improving them to a high level of expertise. This investment might be in the form of budget required to pay for external training, or in terms of time required to build working groups to steer strategy, or run spikes and experiments to increase knowledge and reduce risk. To help with this process we've created the API Maturity Matrix; an Excel-based tool developed to help CxOs & Business Decision Makers assess their organisation's ability to evolve towards an API driven business model. It lists all the main disciplines we think are essential to a successful API Economy programme, your associated maturity level, and what your next actions should be to increase your maturity level. We've used this technique to help a number of customer kickstart their API journey; it will help you too. The Matrix covers the following areas: API Strategy – What's your over-arching strategy for using APIs? How is this going to transform your business? What impact will it have? API Governance – Do you understand the full product lifecycle of an API? Do you have all the governance processes in place to manage this? Legal Considerations – Do you understand the legal obligations and associated commercial risks of providing APIs? API Value Chain / Commercials – Do you understand the value that your APIs deliver? Do you understand the commercial value that can be attributed? Sales & Marketing – How are you going to take your APIs to market? How can you translate your value proposition to specific verticals, sectors & channels? API Documentation – What's the best approach for documenting your APIs? Do you have the correct internal resources? How are you going to co-ordinate content publication with code releases? API Design Principles – Does your organisation have an understanding and appreciation of how to create evolvable, secure, versionable, APIs with the performance and scale requirements demanded by your business model & growth plans? API Development – Does your engineering organisation have the relevant engineering skills to design and build the required APIs? API Quality – Do you understand the quality gates required for building an - API centric platform? Do you have the knowledge, skills & tools required to test them in an automated and repeatable fashion? API Infrastructure – Do you have an appreciation of what is required to support the performance, scale and security of the platform required in order to meet your SLAs? API Operations – Do you have robust monitoring in place to ensure you SLAs are being met, and if there are issues, they can be diagnosed and rectified rapidly? Do you have change controls in place? Developer (Business to Developer) Support – How are you going to provide 1st and 2nd line support to your API consumers to ensure they have a successful experience? https://www.youtube.com/embed/x1VUUHdI9h8 2916 2017-07-28T00:00:00Z yes no https://endjin.com/what-we-think/talks/the-100-year-start-up-embracing-disruption-in-financial-services https://res.cloudinary.com/endjin/image/upload/f_auto/q_80/assets/images/talks/the-100-year-start-up.jpg The 100 Year Start-up - Embracing Disruption in Financial Services Barry Smart (CTO, Hymans Robertson) and Matthew Adams (Co-Founder, endjin) discuss how Hymans Robertson a market leading actuarial consultancy have realigned their business towards innovation, placing FinTech and the cloud at the heart of their offerings in the emerging API, Data, and App economies. Most disruptive start-ups aren't building vertically integrated, full-stack solutions. They connect various pieces of the value chain using public APIs across social media, e-commerce, payment providers, supply chain and more, injecting their special sauce along the way to create new propositions. Hymans Robertson are applying this start-up culture to an established, market leading business. Hymans Robertson was set up in Glasgow in 1921 and is one of the longest established independent firms of consultants and actuaries in the UK, have over 560 staff, including 64 partners, 16 of whom are Members (i.e. share-owning partners) who are all fully involved in the day-to-day management of the Firm. https://www.youtube.com/embed/BmhcpX20uNA 1558 2015-11-11T00:00:00Z yes no