Error generated in Synapse when I query dataverse data in Spark Notebook

Question

Error generated in Synapse when I query dataverse data in Spark Notebook

Robert O'Neill 6

Hey there,

I am having an issue querying the data from the Synapse Link for Dataverse in a Spark Notebook within Synapse.

I am able to run a SQL query against the data (which appears in Synapse as a Lake Database) and it returns data. See below

However when I run a query in Spark Notebook I get the following error:

AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
Traceback (most recent call last):

File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 723, in sql
return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)

File "/home/trusted-service-user/cluster-env/env/lib/python3.8/site-packages/py4j/java_gateway.py", line 1304, in call
return_value = get_return_value(

File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 117, in deco
raise converted from None

pyspark.sql.utils.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException

See Screenshot~:

The Synapse workspace has Owner and Storage Blob Data Contributor access on the storage account.

Anyone have any ideas? I'm really stuck with this one.

HimanshuSinha 19,577 Reputation points Microsoft Employee Moderator

2022-05-18T22:28:42.907+00:00
Hello @Robert O'Neill ,
Thanks for the question and using MS Q&A platform.
As we understand the ask here is how to get around the error while running the notebook scripts ., please do let us know if its not accurate.

Try adding yourself to “Storage Blob Data Contributor”.. as the session run on your context you should have access to the underlaying storage..
Please do let me if you have any queries.
Thanks
Himanshu

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Want a reminder to come back and check responses? Here is how to subscribe to a notification

If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
Robert O'Neill 6 Reputation points

2022-05-19T08:49:25.21+00:00

Thanks for the reply Himanshu.

That's correct I am trying to query the data in Spark Notebook.

I already have Storage Blob Data Contributor access.
Broomfield, Darrien 0 Reputation points

2025-04-26T05:44:01.75+00:00

Did you ever solve this? Having this exact same issue in 2025 still