How can we load a non delimited text file using spark scala and save it as a CSV file where column lengths are given dynamically? - scala

Question

For ex - if we have three columns Name , address and age as schema and a line in file has 92 chars where first 50 are name, next 40 are address and last 2 chars is age, and if these column lengths might vary and will be given dynamically, how to read a file and make it delimited and save it as text file?
Could not get the idea at all

Answer 1

Your question has two parts.
Read file
I assume you have a file named input.txt like this:
1 John 100
2 Jack 200
3 Jonah300
10JJ 400
I also assumed that these are the columns:
// Column Name, Length
val columns = Vector(("id", 2), ("name", 5), ("value", 3))
You should find starting position for each column:
case class ColumnInfo(name: String, length: Int, position: Int)
val columnInfos = columns.tail.foldLeft(Vector(ColumnInfo(columns.head._1, columns.head._2, 1))) { (acc, current) =>
acc :+ ColumnInfo(current._1, current._2, acc.last.position + acc.last.length)
}
This will be the result:
Vector(ColumnInfo(id,2,1), ColumnInfo(name,5,3), ColumnInfo(value,3,8))
Now, you can read and parse this file using this code:
val sparkCols = columnInfos map { columnInfo =>
trim(substring(col("value"), columnInfo.position, columnInfo.length)) as columnInfo.name
}
val df = spark.read
.text("input.txt")
.select(sparkCols: _*)
df.show()
This will be the result:
+---+-----+-----+
| id| name|value|
+---+-----+-----+
| 1| John| 100|
| 2| Jack| 200|
| 3|Jonah| 300|
| 10| JJ| 400|
+---+-----+-----+
Save file
You can save file using this code:
df.repartition(1).write.option("header", true).csv("output.csv")
This will be the result:
id,name,value
1,John,100
2,Jack,200
3,Jonah,300
10,JJ,400