Spark For loop for Large Tables using inner join - pyspark

Question

Create a new table from two existing tables A & B, A has history data for 1 year and B has Data of ID's. I need to join this two tables using Spark where performance is fine as well as loop the data for each and every day or month as business_day is the partition. I cannot consider entire tables as every business day has 30 million each.
Table A - has n number of columns such as ID, Business_Day, Name
Table B - has n number of columns - ID, ID_Code
Table A should join table B using ID=ID and get ID_Code along with other columns of A
insert into output_table
select ID, ID_CODE,Business_Day, Name
from A,B where
A.ID=B.ID
I am not sure how to write For loop for the above, insert script works but for a single day it takes 2 hours and that I need to change business day manually for a year which is impossible, but a loop and other performance steps will help it run very much faster.

Answer 1

Spark SQL Query with Python
Source
from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
import pandas as pd
sc = SparkContext(conf=SparkConf())
sqlContext = SQLContext(sc)
# Table A read and spark create dataframe --> df_A
# df_A = sqlContext.createDataFrame(...)
# Table B read and spark create dataframe --> df_B
# df_B = sqlContext.createDataFrame(...)
# Example:
df1 = sqlContext.createDataFrame(
pd.DataFrame.from_records(
[
[1,12,'Test'],
[2,22,'RD']
],
columns=['ID','ID_CODE','Departman']
))
df2 = sqlContext.createDataFrame(
pd.DataFrame.from_records(
[
[1,'friday','Shan'],
[2,'friday','ramazan'],
[3,'friday','bozkir']
],
columns=['ID','Business_Day','Name']))
### pyspark method SQL
df = df_A.join(df_B,df_B.ID == df_A.ID)
.select('ID_CODE','Business_Day','Name')
### Spark SQL method
df1.registerTempTable('df_A')
df2.registerTempTable('df_B')
df = sqlContext.sql("""
SELECT ID_CODE,Business_Day,Name
FROM (
SELECT *
FROM df_A A LEFT JOIN df_B B ON B.ID = A.ID
) df
""")
""").show()
[In]: df.show()
[Out]:
+-------+------------+-------+
|ID_CODE|Business_Day| Name|
+-------+------------+-------+
| 12| friday| Shan|
| 22| friday|ramazan|
+-------+------------+-------+