A four-part guide to running Spark locally in dev containers — from setup and testing to packaging and testability.
Spark Dev Containers
Working locally with spark dev containers
Running Spark locally in a dev container can significantly improve development feedback loops. This first article explains why, and the rest of the series will show how.
Spark dev containers: running Spark locally
Configure a Docker-based dev container that runs Spark locally — shorten your inner dev loop and make automated tests practical with the right base image.
Spark dev containers: writing tests
Having seen earlier in the series how to configure a dev container to run Spark locally, this post shows how to write tests that use that local Spark service.
Spark dev containers: packaging code for testability
Once you've thoroughly tested your code against the local Spark service in your dev container, you'll want to run it in a real Spark cluster. This posts shows how to deploy such code to Microsoft Fabric.