How to parse delimited fields with some (sub)fields empty? - scala

Question

I use Spark 2.1.1 and Scala 2.11.8 in spark-shell.
My input dataset is something like :
2017-06-18 00:00:00 , 1497769200 , z287570731_serv80i:7:175 , 5:Re
2017-06-18 00:00:00 , 1497769200 , p286274731_serv80i:6:100 , 138
2017-06-18 00:00:00 , 1497769200 , t219420679_serv37i:2:50 , 5
2017-06-18 00:00:00 , 1497769200 , v290380588_serv81i:12:800 , 144:Jo
2017-06-18 00:00:00 , 1497769200 , z292902510_serv83i:4:45 , 5:Re
2017-06-18 00:00:00 , 1497769200 , v205454093_serv75i:5:70 , 50:AK
It is saved as a CSV file which is read using sc.textFile("input path")
After a few transformations, this is the output of the RDD I have:
(String, String) = ("Re ",7)
I get this by executing
val tid = read_file.map { line =>
val arr = line.split(",")
(arr(3).split(":")(1), arr(2).split(":")(1))
}
My input RDD is:
( z287570731_serv80i:7:175 , 5:Re )
( p286274731_serv80i:6:100 , 138 )
( t219420679_serv37i:2:50 , 5 )
( v290380588_serv81i:12:800 , 144:Jo )
( z292902510_serv83i:4:45 , 5:Re )
As it can be observed, in the first entry column 2, I have
5:Re
of which I'm getting the output
("Re ",7)
However when I reach the second row, according to the format, column 2 is 138 which should be
138:null
but gives ArrayIndexOutOfBoundsException on executing
tid.collect()
How can I correct this so that null is displayed with 138 and 5 for the second and third rows respectively? I tried to do it this way:
tid.filter(x => x._1 != null )

Answer 1

The problem is that you expect at least two parts in the position while you may have only one.
The following is the line that causes the issue.
{var arr = line.split(","); (arr(3).split(":")(1),arr(2).split(":")(1))});
After you do line.split(",") you then arr(3).split(":")(1) and also arr(2).split(":")(1).
There's certainly too much assumption about the format and got beaten by missing values.
but gives ArrayIndexOutOfBoundsException on executing
That's because you access 3 and 2 elements but have only 2 (!)
scala> sc.textFile("input.csv").
map { line => line.split(",").toSeq }.
foreach(println)
WrappedArray(( z287570731_serv80i:7:175i , 5:Re ))
WrappedArray(( p286274731_serv80i:6:100 , 138 ))
The problem has almost nothing to do with Spark. It's a regular Scala problem where the data is not where you expect it.
scala> val arr = "hello,world".split(",")
arr: Array[String] = Array(hello, world)
Note that what's above is just a pure Scala.
Solution 1 - Spark Core's RDDs
Given the following dataset...
2017-06-18 00:00:00 , 1497769200 , z287570731_serv80i:7:175 , 5:Re
2017-06-18 00:00:00 , 1497769200 , p286274731_serv80i:6:100 , 138
2017-06-18 00:00:00 , 1497769200 , t219420679_serv37i:2:50 , 5
2017-06-18 00:00:00 , 1497769200 , v290380588_serv81i:12:800 , 144:Jo
2017-06-18 00:00:00 , 1497769200 , z292902510_serv83i:4:45 , 5:Re
2017-06-18 00:00:00 , 1497769200 , v205454093_serv75i:5:70 , 50:AK
...I'd do the following:
val solution = sc.textFile("input.csv").
map { line => line.split(",") }.
map { case Array(_, _, third, fourth) => (third, fourth) }.
map { case (third, fourth) =>
val Array(_, a # _*) = fourth.split(":")
val Array(_, right, _) = third.split(":")
(a.headOption.orNull, right)
}
scala> solution.foreach(println)
(Re,7)
(null,6)
(Re,4)
(null,2)
(AK,5)
(Jo,12)
Solution 2 - Spark SQL's DataFrames
I strongly recommend using Spark SQL for such data transformations. As you said, you are new to Spark, so why not start from the right place which is exactly Spark SQL.
val solution = spark.
read.
csv("input.csv").
select($"_c2" as "third", $"_c3" as "fourth").
withColumn("a", split($"fourth", ":")).
withColumn("left", $"a"(1)).
withColumn("right", split($"third", ":")(1)).
select("left", "right")
scala> solution.show(false)
+----+-----+
|left|right|
+----+-----+
|Re |7 |
|null|6 |
|null|2 |
|Jo |12 |
|Re |4 |
|AK |5 |
+----+-----+

Answer 2

If your data is as below in a file
( z287570731_serv80i:7:175 , 5:Re )
( p286274731_serv80i:6:100 , 138 )
( t219420679_serv37i:2:50 , 5 )
( v290380588_serv81i:12:800 , 144:Jo )
( z292902510_serv83i:4:45 , 5:Re )
Then you can use
val tid = sc.textFile("path to the input file")
.map(line => line.split(","))
.map(array => {
if (array(1).contains(":")) (array(1).split(":")(1).replace(")", "").trim, array(0).split(":")(1))
else (null, array(0).split(":")(1))
})
tid.foreach(println)
which should give you output as
(Re,7)
(null,6)
(null,2)
(Jo,12)
(Re,4)
But if you have data as
2017-06-18 00:00:00 , 1497769200 , z287570731_serv80i:7:175 , 5:Re
2017-06-18 00:00:00 , 1497769200 , p286274731_serv80i:6:100 , 138
2017-06-18 00:00:00 , 1497769200 , t219420679_serv37i:2:50 , 5
2017-06-18 00:00:00 , 1497769200 , v290380588_serv81i:12:800 , 144:Jo
2017-06-18 00:00:00 , 1497769200 , z292902510_serv83i:4:45 , 5:Re
2017-06-18 00:00:00 , 1497769200 , v205454093_serv75i:5:70 , 50:AK
2017-06-18 00:00:00 , 1497769200 , z287096299_serv80i:19:15000 , 39:Re
Then you need to do
val tid = sc.textFile("path to the input file")
.map(line => line.split(","))
.map(array => {
if (array(3).contains(":")) (array(3).split(":")(1).replace(")", "").trim, array(2).split(":")(1))
else (null, array(2).split(":")(1))
})
tid.foreach(println)
And you should have output as
(Re,7)
(null,6)
(null,2)
(Jo,12)
(Re,4)
(AK,5)
(Re,19)

Answer 3

ArrayIndexOutOfBounds is occurring because the element will not be there if no : is present in the second element of the tuple.
You can check if : is present in the second element of each tuple. And then use map to give you an intermediate RDD on which you can run your current query.
val rdd = sc.parallelize(Array(
( "z287570731_serv80i:7:175" , "5:Re" ),
( "p286274731_serv80i:6:100" , "138" ),
( "t219420679_serv37i:2:50" , "5" ),
( "v290380588_serv81i:12:800" , "144:Jo" ),
( "z292902510_serv83i:4:45" , "5:Re" )))
rdd.map { x =>
val idx = x._2.lastIndexOf(":")
if(idx == -1) (x._1, x._2+":null")
else (x._1, x._2)
}
There are obviously better (lesser lines of code) ways to do what you want to accomplish but as a beginner, it's good to layout each step in a single command so t's easily readable and understandable, specially with scala where you can stop global warming with a single line of code.