Connecting Microsoft SQL Server using pyspark, throws Error: - pyspark

Question

Please guide me the steps to connect and read data from MS SQL by using Pyspark.
Below is my code and the error message that i am getting when i am trying to load data from MS SQL Server. Please guide me.
import urllib
import findspark
findspark.init()
from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
APP_NAME = 'My Spark Application'
conf = SparkConf().setAppName("APP_NAME").setMaster("local[4]")
sc = SparkContext(conf=conf)
sqlcontext = SQLContext(sc)
jdbcDF = sqlcontext.read.format("jdbc")\
.option("url", "jdbc:sqlserver:XXXX:1433")\
.option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver")\
.option("dbtable", "dbo.XXXX")\
.option("user", "XXXX")\
.option("password", "XXX")\
.load()
******************************ERROR***************************************
teway.py", line 1133, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "C:\spark-2.0.1-bin-hadoop2.6\python\pyspark\sql\utils.py", line 63, in d
eco
return f(*a, **kw)
File "C:\spark-2.0.1-bin-hadoop2.6\python\lib\py4j-0.10.3-src.zip\py4j\protoco
l.py", line 319, in get_return_value
format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o66.load.
: java.lang.NullPointerException
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable
(JDBCRDD.scala:167)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.<init>(J
DBCRelation.scala:117)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.
createRelation(JdbcRelationProvider.scala:53)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation
(DataSource.scala:330)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:149)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:122)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:745)

Answer 1

The following solution worked for me:
Include mssql-jdbc-7.0.0.jre8.jar file into jars sub-folder( ex: C:\spark\spark-2.2.2-bin-hadoop2.7\jars) or you can paste any of the jar file based on your system.
Then use the following command to connect to MS SQL server & create the Spark Dataframe:
dbData = spark.read.jdbc("jdbc:sqlserver://servername;databaseName:ExampleDB;user:username;password:password","tablename")

Answer 2

Download mssql-jdbc-x.x.x.jrex.jar file (https://learn.microsoft.com/en-us/sql/connect/jdbc/download-microsoft-jdbc-driver-for-sql-server?view=sql-server-ver15)
Run the following code:
import findspark
findspark.init()
from pyspark import SparkContext, SparkConf, SQLContext
appName = "PySpark SQL Server Example - via JDBC"
master = "local[*]"
conf = SparkConf() \
.setAppName(appName) \
.setMaster(master) \
.set("spark.driver.extraClassPath","path/to/mssql-jdbc-x.x.x.jrex.jar")
sc = SparkContext.getOrCreate(conf=conf)
sqlContext = SQLContext(sc)
spark = sqlContext.sparkSession
database = "mydatabase"
table = "dbo.mytable"
user = "username"
password = "password"
jdbcDF = spark.read.format("jdbc") \
.option("driver" , "com.microsoft.sqlserver.jdbc.SQLServerDriver") \
.option("url", f"jdbc:sqlserver://serverip:1433;databaseName={database}") \
.option("dbtable", "mytable") \
.option("user", user) \
.option("password", password) \
.load()
jdbcDF.show()