Create pyspark data frame from list? - pyspark

Question

So I have a list of headers e.g.
Headers=["col1", "col2", "col3"]
and a list of rows
Body=[ ["val1", "val2", "val3"], ["val1", "val2", "val3"] ]
where val1 corresponds to a value that should go under col1 ect.
If I try createDataFrame(data=Body) it gives an error cant infer schmea type for str
Is it possible to get a list like this into a pyspark dataframe?
I have tried appending the header to the body e.g.
body.append(header) and then using the create data frame function but it throws up this error:
field _22: Can not merge type <class 'pyspark.sql.types.DoubleType'> and <class 'pyspark.sql.types.LongType'>
This is my whole code for this part for generating the body and header:
Basically I using openpyxl to read an excel file, where it skips the first x rows ect and only reads in sheets which have certain column names.
After the body and header has been generated I want to read it directly into spark.
We had a contractor who wrote it out as a csv and then read it in using spark but just seemed to make more sense to directly put it into spark.
I want to the columns to all be strings at this point
import csv
from os import sys
excel_file = "/dbfs/{}".format(path)
wb = load_workbook(excel_file, read_only=True)
sheet_names = wb.get_sheet_names()
sheets = spark.read.option("multiline", "true").format("json").load(configPath)
if dataFrameContainsColumn(sheets, "sheetNames"):
config_sheets = jsonReader(configFilePath,"sheetNames")
else:
config_sheets= []
skip_rows=-1
#get a list of the required columns
required_fields_list = jsonReader(configFilePath,"requiredColumns")
for worksheet_name in sheet_names:
count=0
sheet_count=0
second_break=False
worksheet = wb.get_sheet_by_name(worksheet_name)
#assign the sheet name to the object sheet
#create empty header and body lists for each sheet
header = []
body = []
#for each row in the sheet we need to append the cells to the header and body
for i,row in enumerate(worksheet.iter_rows()):
#if the row index is greater then skip rows then we want to read that row in as the header
if i==skip_rows+1:
header.append([cell.value for cell in row])
elif i>skip_rows+1:
count=count+1
if count==1:
header=header[0]
header = [w.replace(' ', '_') for w in header]
header = [w.replace('.', '') for w in header]
if(all(elem in header for elem in required_fields_list)==False):
second_break=True
break
else:
count=2
sheet_count=sheet_count+1
body.append([cell.value for cell in row])```

Answer 1

There are a few method to create a dataframe from a list.
You can check them out here
Let Spark infer the schema
list_of_persons = [('Arike', 28, 78.6), ('Bob', 32, 45.32), ('Corry', 65, 98.47)]
df = sc.parallelize(list_of_persons).toDF(['name', 'age', 'score'])
df.printSchema()
df.show()
root
|-- name: string (nullable = true)
|-- age: long (nullable = true)
|-- score: double (nullable = true)
+-----+---+-----+
| name|age|score|
+-----+---+-----+
|Arike| 28| 78.6|
| Bob| 32|45.32|
|Corry| 65|98.47|
+-----+---+-----+
Specify the types using a map transformation
list_of_persons = [('Arike', 28, 78.6), ('Bob', 32, 45.32), ('Corry', 65, 98.47)]
rdd = sc.parallelize(list_of_persons)
person = rdd.map(lambda x: Row(name=x[0], age=int(x[1]), score=float(x[2])))
schemaPeople = sqlContext.createDataFrame(person)
schemaPeople.printSchema()
schemaPeople.show()
root
|-- age: long (nullable = true)
|-- name: string (nullable = true)
|-- score: double (nullable = true)
+---+-----+-----+
|age| name|score|
+---+-----+-----+
| 28|Arike| 78.6|
| 32| Bob|45.32|
| 65|Corry|98.47|
+---+-----+-----+