Spark

Write data back to SharePoint from Synapse Notebooks using PySpark, the Microsoft Graph API, and Service Principal auth — Drive IDs, tokens, and upload patterns.

This post describes an approach to copy files and data from SharePoint into Azure using Synapse Notebooks.

Once you've thoroughly tested your code against the local Spark service in your dev container, you'll want to run it in a real Spark cluster. This posts shows how to deploy such code to Microsoft Fabric.

Having seen earlier in the series how to configure a dev container to run Spark locally, this post shows how to write tests that use that local Spark service.

Configure a Docker-based dev container that runs Spark locally — shorten your inner dev loop and make automated tests practical with the right base image.