Skip to content

Data

There's something wrong with the Pandas API on Spark

There's something wrong with the Pandas API on Spark

Carmel Eve

Fix the following issues: Errors converting large datasets to pandas, pandas for Spark is very slow, and pandas for Spark column reduction doesn't reduce data.
Per-Property Rows from JSON in Spark on Microsoft Fabric

Per-Property Rows from JSON in Spark on Microsoft Fabric

Ian Griffiths

Spark doesn't always interpret JSON how we'd like. For example, if each key/value pair in a JSON object is conceptually one item, Spark won't give you a row per item by default. This article shows how to nudge Spark in the right direction.