A four-part guide to running Spark locally in dev containers — from setup and testing to packaging and testability.
Spark Dev Containers
Working locally with spark dev containers
Running Spark locally in a dev container can significantly improve development feedback loops. This first article explains why, and the rest of the series will show how.
Spark dev containers: running Spark locally
See how to configure a dev container to run Spark locally, to improve development feedback loops.
Spark dev containers: writing tests
Having seen earlier in the series how to configure a dev container to run Spark locally, this post shows how to write tests that use that local Spark service.
Spark dev containers: packaging code for testability
Once you've thoroughly tested your code against the local Spark service in your dev container, you'll want to run it in a real Spark cluster. This posts shows how to deploy such code to Microsoft Fabric.