Creating Dataframe out of two different arrays spark - scala

Question

I am using spark streaming and am creating this dataframe from the kafka message:
|customer|initialLoadComplete|initialLoadRunning| messageContent| tableName|
+--------+-------------------+------------------+--------------------+-----------------+
| A| false| true|TEFault_IdReason...|Timed_Event_Fault|
| A| false| true|TEFault_IdReason...|Timed_Event_Fault|
+--------+-------------------+------------------+--------------------+-----------------+
Now I want to extract out messageContent, messageContent is basically like a CSV that includes the raw data and the first line is the columns.
I can extract out the headers in the following way from the messageContent field.
val Array1 = ssc.sparkContext.parallelize(rowD.getString(2).split("\u0002")(0))
So Array1 looks like this:
Array1: col1^Acol2^Acol3
Array2 is basically the raw data, each column value seperated by ^A and record seperated by ^B.
^A is a column seperator. ^B is record seperator
So this is what array2 could look like:
Array2 = value1^Avalue2^Avalue3^Bvalue4^Avalue5^Avalue6^Bvalue7^Avalue8^Avalue9
Basically I want to create a dataframe out of this so it looks like this:
col1 | col2 | col3
-------------------------
value1 | value2 | value3
value4 | value5 | value6
value7 | value8 | value9
^B is the record delimiter.
When we were reading from a hdfs file, we created a dataframe via this command:
val df = csc.read.format("com.databricks.spark.csv").option("header", "true").option("inferSchema", "true").option("delimiter", "\u0001").load(hdfsFile)
But this time I am creating a dataframe from two arrays from memory. Array1 is the headers for the values in array2 and array2 is record seperated by ^B.
What could be the equivalent of creating a dataframe in this approach as I did for creating a dataframe from a file.

Answer 1

I am inferring the following from your question.
Array1 is a rdd of only one entry col1^Acol2^Acol3
Array2 is a rdd with each entry looking something like this. value1^Avalue2^Avalue3^Bvalue4^Avalue5^Avalue6^Bvalue7^Avalue8^Avalue9
with these assumptions in place the following should work.
val array1 = sc.parallelize(Seq("col1\u0002col2\u0002col3"))
val array2 = sc.parallelize(Seq("value1\u0001value2\u0001value3\u0002value4\u0001value5\u0001value6\u0002value7\u0001value8\u0001value9"))
val data = array2.flatMap(x => x.split("\u0002")).map(x => x.split('\u0001')).collect()
val result = array2
.flatMap(x => x.split("\u0002"))
.map(x => x.split('\u0001'))
.map({ case Array(x,y,z) => (x,y,z)})
.toDF(array1.flatMap(x => x.split('\u0002')).collect(): _*)
result.show()
+------+------+------+
| col1| col2| col3|
+------+------+------+
|value1|value2|value3|
|value4|value5|value6|
|value7|value8|value9|
+------+------+------+