How to read .dat file using pyspark.sql.session.SparkSession object - pyspark

Question

I am a newbie to Spark, bear my silly mistakes if there's any (Open for your suggestions :))
I have created a pyspark.sql.session.SparkSession object using following code:
from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local[*]").getOrCreate()
I know that I can read a csv file using spark.read.csv('filepath').
Now, I would like to read .dat file using that SparkSession object.
My ratings.dat file looks like:
1::1193::5::978300760
1::661::3::978302109
1::914::3::978301968
1::3408::4::978300275
My code:
ratings = spark.read.format('dat').load('filepath/ratings', sep='::')
Output:
An error occurred while calling o102.load.
: java.lang.ClassNotFoundException
Expected Output:
+------+-------+------+---------+
|UserID|MovieID|Rating|Timestamp|
+------+-------+------+---------+
| 1| 1193| 5|978300760|
| 1| 661| 3|978302109|
| and.........so.......on.......|
+------+-------+------+---------+
Note: My ratings.dat file do not contain headers and separator is ::.
Questions:
How can I read .dat file?
How can I add my custom header like I mentioned in Expected output?
So, How can I achieve my expected output? Where am I committing mistakes?
I would love to read your suggestions and answers :)
Would really appreciate long and detailed answers.

Answer 1

You can just use the csv reader with :: as the separator, and provide a schema:
df = spark.read.csv('ratings.dat', sep='::', schema='UserID int, MovieID int, Rating int, Timestamp long')
df.show()
+------+-------+------+---------+
|UserID|MovieID|Rating|Timestamp|
+------+-------+------+---------+
| 1| 1193| 5|978300760|
| 1| 661| 3|978302109|
| 1| 914| 3|978301968|
| 1| 3408| 4|978300275|
+------+-------+------+---------+