Custom Scala Spark Jobs in Azure Synapse | endjin

16th June 2020 · 13 min watch

By Ian Griffiths Technical Fellow I

Learn to run a Scala custom Spark job in Azure Synapse: create a workspace, add a Spark pool, configure storage, write code in VS Code, and execute.

About this talk

Tutorial

Azure Synapse workspaces can host a Spark cluster. In addition to providing the execution environment for certain Synapse features such as Notebooks, you can also write custom code that runs as a job inside Synapse hosted Spark cluster.

This video walks through the process of running a Scala custom Spark job in Azure Synapse. It shows how to create the Synapse workspace in the Azure portal, how to add a Spark pool, and how to configure a suitable storage account. It also shows how to write the custom job in Scala, using Visual Studio Code's support for Docker-based development to target the exact JDK and Scala version used in Synapse. You will see how to upload the built output to Azure, and then how to configure Azure Synapse to execute the resulting JAR as a custom job.

About the presenter

Ian Griffiths

Ian has worked across an extraordinary breadth of computing - from embedded real-time systems and broadcast television to medical imaging and cloud-scale architectures. As Technical Fellow at endjin, he brings this deep cross-domain experience to bear on the hardest technical problems.

A 17-time Microsoft MVP in Developer Technologies, Ian is the author of O'Reilly's Programming C# 12.0 and one of the foremost authorities on the C# language and high-performance .NET development. He's a maintainer of Reactive Extensions for .NET, Reaqtor, and endjin's 50+ open source projects.

Ian has created Pluralsight courses on WPF fundamentals, WPF advanced topics, WPF v4, and the TPL, and has given over 20 talks at conferences worldwide. Technology brings him joy.