How to create DataFrame schema from schema json file in pyspark? - pyspark

Question

I'm trying to use Pyspark to create DataFrame schema from schema json file. Once DataFrame schema created, I will load json data files by using this schema. Could somebody help me? Thanks in advance. For my schema json file look like below:
[
{
"name": "visitorId",
"type": "INTEGER",
"mode": "NULLABLE"
},
{
"name": "visitStartTime",
"type": "INTEGER",
"mode": "NULLABLE"
},
{
"name": "totals",
"type": "RECORD",
"mode": "NULLABLE",
"fields": [
{
"name": "visits",
"type": "INTEGER",
"mode": "NULLABLE"
},
{
"name": "hits",
"type": "INTEGER",
"mode": "NULLABLE"
},
{
"name": "pageviews",
"type": "INTEGER",
"mode": "NULLABLE"
},
{
"name": "transactions",
"type": "INTEGER",
"mode": "NULLABLE"
},
{
"name": "timeOnScreen",
"type": "INTEGER",
"mode": "NULLABLE"
}
]
},
{
"name": "channelGrouping",
"type": "STRING",
"mode": "NULLABLE"
}
]

Answer 1

Your schema is not defined as expected and pyspark cannot parse it.
I've changed your schema to:
{
"type": "struct",
"fields": [
{
"name": "visitorId",
"type": "integer",
"nullable": true,
"metadata": {}
},
{
"name": "visitStartTime",
"type": "integer",
"nullable": true,
"metadata": {}
},
{
"name": "totals",
"type": {
"type": "struct",
"fields": [
{
"name": "visits",
"type": "integer",
"nullable": true,
"metadata": {}
},
{
"name": "hits",
"type": "integer",
"nullable": true,
"metadata": {}
},
{
"name": "pageviews",
"type": "integer",
"nullable": true,
"metadata": {}
},
{
"name": "transactions",
"type": "integer",
"nullable": true,
"metadata": {}
},
{
"name": "timeOnScreen",
"type": "integer",
"nullable": true,
"metadata": {}
}
]
},
"nullable": true,
"metadata": {}
},
{
"name": "channelGrouping",
"type": "string",
"nullable": true,
"metadata": {}
}
]
}
Saved that as schema.json file and then created a StructType from this json using:
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType
import json
if __name__ == "__main__":
spark = SparkSession \
.builder \
.appName("Test") \
.getOrCreate()
with open("schema.json") as f_in:
schema_data = json.load(f_in)
data = {}
schemaFromJson = StructType.fromJson(schema_data)
df = spark.createDataFrame(spark.sparkContext.parallelize(data), schemaFromJson)
df.printSchema()
The result is:
root
|-- visitorId: integer (nullable = true)
|-- visitStartTime: integer (nullable = true)
|-- totals: struct (nullable = true)
| |-- visits: integer (nullable = true)
| |-- hits: integer (nullable = true)
| |-- pageviews: integer (nullable = true)
| |-- transactions: integer (nullable = true)
| |-- timeOnScreen: integer (nullable = true)
|-- channelGrouping: string (nullable = true)