Skip to content
Ed Freeman By Ed Freeman Software Engineer II
Azure Databricks CLI "Error: JSONDecodeError: Expecting property name enclosed in double quotes:..."

Have you been trying to create a Databricks cluster using the CLI? Have you been getting infuriated by something seemingly so trivial? Well, join the club. Although, get ready to depart the club because I may have the solution you need.

When creating a cluster using the CLI command databricks clusters create, you're required to pass in either a JSON string or a path to a JSON file. I recently opted for the first option. [Note: I'm using PowerShell to talk to the Databricks CLI.]

How it transpired

Here's what (I feel) should have worked.

But running that, we receive the output:

Error: JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 5 (char 7)

Huh? You asked for a JSON string, I gave you a JSON string. Why are you complaining? Maybe the Databricks CLI wants me to wrap more quotes around my --json argument. Not sure why, but let's try that.

databricks clusters create --json "$configToJson"

Nope. Same error. More quotes?

databricks clusters create --json "'$configToJson'"

Of course, there's no chance that's going to work and it gives us the error:

Error: JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Hmm. What could be happening?

The best hour you can spend to refine your own data strategy and leverage the latest capabilities on Azure to accelerate your road map.

A little bit of Googling later, I find someone who had the same problem as me (who was using the windows command prompt). They alluded to the need to "escape" the double quotes with a backslash within the JSON string. Sounds odd (a backslash is neither the escape character in PowerShell nor the Windows Command Prompt), but let's give it a whirl:

And hey presto, it works:

{ "cluster_id": "0704-090525-blocs355" }

But... why?

At this point, it's very easy to shrug this off now that it's working and not bother to try and understand why it's now working. I mentioned that "a backslash is neither the escape character in PowerShell nor the Windows Command Prompt", but there are numerous common languages for which it is the escape character. After locating the Databricks CLI GitHub repo, I saw that it was written in Python, a language which uses the backslash as an escape character.

Azure Weekly is a summary of the week's top Microsoft Azure news from AI to Availability Zones. Keep on top of all the latest Azure developments!

Under the covers, the Databricks CLI is using the json.loads() method to parse our --json argument, and the error we're getting is a JSONDecodeError coming from that json package. The loads() method takes a string in the form '{ "name":"John", "age":30, "city":"New York"}' and converts it to a Python dictionary. This should work with the argument we used in the first attempt earlier, but it doesn't. My hypothesis (which I'm unsure how to prove) is that the JSON string argument is implictly being wrapped in double quotes in the process of being passed to the Python method, so the method receives this:

which isn't a valid representation of a string in Python (because the 2nd double quote closes the first double quote), and cannot be parsed into a dictionary. Add in the interior double quote escaping, and all's good - we now have a string that Python can understand.

A sensible alternative

I mentioned earlier that you can also pass in a path to a JSON file using the --json-file parameter. This works exactly how you'd expect (no funny business with escaping quotes since you're passing it as a .json file as opposed to a JSON string). In our case, however, having to define the JSON object elsewhere in our code-base would have been sub-optimal, but YMMV.

I hope this blog has helped some of you!

Ed Freeman

Software Engineer II

Ed Freeman

Ed is a Data Engineer helping to deliver projects for clients of all shapes and sizes, providing best of breed technology solutions to industry specific challenges. He focusses primarily on cloud technologies, data analytics and business intelligence, though his Mathematical background has also led to a distinct interest in Data Science, Artificial Intelligence, and other related fields.

He also curates a weekly newsletter, Power BI Weekly, where you can receive all the latest Power BI news, for free.

Ed won the Cloud Apprentice of the Year at the Computing Rising Star Awards 2019.