PySpark execute all test cases in a directory - pyspark

Question

I am trying to develop a script which will run all the spark sql queries kept in a directory.
I have been able to do that in Python, but pyspark is a different game.
Below is the python script I use to read and execute all query files in a directory.
import sys,csv,sqlite3,codecs,unicodedata, string,glob, os,c
conn=psycopg2.connect(database="xxx", user="xxxx", password="xxxx",
host="localhost", port="5432") cur = conn.cursor() print("done")
with open("*.txt", "r") as ins: for line in ins:
words=line.split('|') print(words) query=words[0]
pmicode=words[1] print(query)
cur = conn.cursor()
cur.execute(query) conn.commit() conn.close()
Is it possible to replicate this in PySpark ?
Thanks,
Pankaj

Answer 1

I'm guessing you want pyspark to pull data from the postgres database that you're using in this python script.
If the current code in Python is something like:
import sys, csv, sqlite3, codecs, unicodedata, string, glob, os
conn = psycopg2.connect(database="xxx", user="xxxx", password="xxxx", host="localhost", port="5432")
cur = conn.cursor()
print("done")
def runSQL(query):
cur = conn.cursor()
cur.execute(query)
conn.commit()
with open("*.txt", "r") as ins:
for line in ins:
words = line.split('|')
print(words)
query = words[0]
pmicode = words[1]
print(query)
conn.close()
The equivalent would be to use JDBC to connect and execute command with a sqlContext:
import sys, csv, sqlite3, codecs, unicodedata, string, glob, os
postgres_url = 'jdbc:postgresql://localhost:5432/database'
properties = {"user": "xxxx", "password": "xxxx"}
print("done")
def runSQL(query):
return sqlContext.read.jdbc(
url=postgres_url,
table="( {0} ) TEMPDB_SPARK_DELINQ".format(query)
with open("*.txt", "r") as ins:
for line in ins:
words = line.split('|')
print(words)
query = words[0]
pmicode = words[1]
print(query)
runSQL(query)