Saving and Overwriting a file in Spark Scala - scala

Question

I have a text file where my first column is represented with table name and the second column is represented with date. The delimiter between two columns is represented by space. The data is represented as follows
employee.txt
organization 4-15-2018
employee 5-15-2018
My requirement is to read the file and update the date column based on the business logic and save/overwrite the file. Below is my code
object Employee {
def main(args: Array[String]) {
val conf = new SparkConf().setMaster("local").setAppName("employeedata")
val sc = new SparkContext(conf)
var input = sc.textFile("D:\\employee\\employee.txt")
.map(line => line.split(' '))
.map(kvPair => (kvPair(0), kvPair(1)))
.collectAsMap()
//Do some operations
// Do iteration and update the hashmap as follows
val finalMap = input + (tableName -> updatedDate)
sc.stop()
}
How to save/overwrite(if exists) the finalMap in the above scenario?

Answer 1

My requirement is to read the file and update the date column based on the business logic and save/overwrite the file.
Never do something like this directly. Always:
Write data to a temporary storage first.
Delete original using standard file system tools.
Rename temporary output using standard file system tools.
An attempt to overwrite data directly will, with high probability, result in a partial or complete data loss.