Best approch for parsing large structured file with Apache spark - scala

Question

I have huge text file (in GBs) with plan text data in each line, which needs to be parsed and extracted to a structure for further processing. Each line has text with 200 charactor length and I have an Regular Expression to parse each line and split into different groups, which will later saved to a flat column data
data sample
1759387ACD06JAN1910MAR191234567ACRT
RegExp
(.{7})(.{3})(.{7})(.{7})(.{7})(.{4})
Data Structure
Customer ID, Code, From Date, To Date, TrasactionId, Product code
1759387, ACD, 06JAN19, 10MAR19, 1234567, ACRT
Please suggest a BEST approch to parse this huge data and push to In Memory grid, which will be used again by Spark Jobs for further processing, when respective APIs are invoked.

Answer 1

You can use the DF approach. Copy the serial file to HDFS using -copyFromLocal command
and use the below code to parse each records
I'm assuming the sample records in gireesh.txt as below
1759387ACD06JAN1910MAR191234567ACRT
2759387ACD08JAN1910MAY191234567ACRY
3759387ACD03JAN1910FEB191234567ACRZ
The spark code
import org.apache.log4j.{Level, Logger}
import org.apache.spark._
import org.apache.spark.sql._
import org.apache.spark.sql.Encoders._
import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
object Gireesh {
def main(args: Array[String]) {
Logger.getLogger("org").setLevel(Level.ERROR)
val spark = SparkSession.builder().appName("Operations..").master("local[*]").getOrCreate()
import spark.implicits._
val pat="""(.{7})(.{3})(.{7})(.{7})(.{7})(.{4})""".r
val headers = List("custid","code","fdate","tdate","tranid","prdcode")
val rdd = spark.sparkContext.textFile("in/gireesh.txt")
.map( x => {
val y = scala.collection.mutable.ArrayBuffer[String]()
pat.findAllIn(x).matchData.foreach( m=> y.appendAll(m.subgroups))
(y(0).toLong,y(1),y(2),y(3),y(4).toLong,y(5))
}
)
val df = rdd.toDF(headers:_*)
df.printSchema()
df.show(false)
}
}
gives the below results.
root
|-- custid: long (nullable = false)
|-- code: string (nullable = true)
|-- fdate: string (nullable = true)
|-- tdate: string (nullable = true)
|-- tranid: long (nullable = false)
|-- prdcode: string (nullable = true)
+-------+----+-------+-------+-------+-------+
|custid |code|fdate |tdate |tranid |prdcode|
+-------+----+-------+-------+-------+-------+
|1759387|ACD |06JAN19|10MAR19|1234567|ACRT |
|2759387|ACD |08JAN19|10MAY19|1234567|ACRY |
|3759387|ACD |03JAN19|10FEB19|1234567|ACRZ |
+-------+----+-------+-------+-------+-------+
EDIT1:
You can have the map "transformation" in a separate function like below.
def parse(record:String) = {
val y = scala.collection.mutable.ArrayBuffer[String]()
pat.findAllIn(record).matchData.foreach( m=> y.appendAll(m.subgroups))
(y(0).toLong,y(1),y(2),y(3),y(4).toLong,y(5))
}
val rdd = spark.sparkContext.textFile("in/gireesh.txt")
.map( x => parse(x) )
val df = rdd.toDF(headers:_*)
df.printSchema()

Answer 2

You need to tell spark which file to read and how to process the content while reading it.
Here is an example:
val numberOfPartitions = 5 // this needs to be optimized based on the size of the file and the available resources (e.g. memory)
val someObjectsRDD: RDD[SomeObject] =
sparkContext.textFile("/path/to/your/file", numberOfPartitions)
.mapPartitions(
{ stringsFromFileIterator =>
stringsFromFileIterator.map(stringFromFile => //here process the raw string and return the result)
}
, preservesPartitioning = true
)
In the code snippet SomeObject is an object with the data structure from the question