Execute queries on redshift using Pyspark - pyspark

Question

Can any one of you suggest the method of executing queries on redshift tables using pyspark?

Answer 1

Using pyspark data-frames option is one option to read/write to Redshift tables. And executing queries with dataframe's help is restricted to using preactions/postactions method.
If you need to execute multiple queries, one way is to use psycopg2 module.
First, you need to install psycopg2 module on your server:
sudo python -m pip install psycopg2
Then open the pyspark shell and execute the following:
import psycopg2
conn={'db_name': {'hostname':'redshift_host_url','database':'database_on_redshift','username':'redshift_username','password':'p', 'port':your_port} }
db='db_name'
hostname, username, password, database, portnumber = conn[db]['hostname'], conn[db]['username'], conn[db]['password'], conn[db]['database'], conn[db]['port']
con = psycopg2.connect( host=hostname, port=portnumber, user=username, password=password, dbname=database)
query="insert into sample_table select * from table1"
cur = con.cursor()
rs = cur.execute(query)
con.commit()
con.close()
You can also refer: https://www.psycopg.org/docs/usage.html