Scala ElasticSearch indexing in relation to dynamically changing schema - scala

Question

I found the following code below on CERN's website.
FYI: I am using spark 1.3
The example code is fantastic when you know the schema of the dataset you want to index to elasticsearch.
However, could somebody point me in the right direction so that I can create a method as follows:
Pass in as an argument the schema structure from external source (col name / datatype) (the hard bit) along with file name to be indexed (easy bit)?
Perform the schema mappings inside the function dynamically.
By having a method like this would allow me to generate a mapped and indexed dataset in ES.
Example Code:
//import elasticsearch packages
import org.elasticsearch.spark._
//define the schema
case class MemT(dt: String (link is external), server: String (link is external), memoryused: Integer (link is external))
//load the csv file into rdd
val Memcsv = sc.textFile("/tmp/flume_memusage.csv")
//split the fields, trim and map it to the schema
val MemTrdd = Memcsv.map(line=>line.split(",")).map(line=>MemT(line(0).trim.toString,line(1).trim.toString,line(2).trim.toInt))
//write the rdd to the elasticsearch
MemTrdd.saveToEs("fmem/logs")
Thank you!
source:
https://db-blog.web.cern.ch/blog/prasanth-kothuri/2016-05-integrating-hadoop-and-elasticsearch-%E2%80%93-part-2-%E2%80%93-writing-and-querying

Answer 1

What i wanted to achieve is to be able index directly into ES from a DataFrame,.
I required that the index mappings be driven from external schema source. Here is how I achieved that....
BTW: There is additional validation/processing that I have omitted, but this skeleton code should get those who require similar needs going....
I have included the following ES dependency in my build.sbt file
"org.elasticsearch" % "elasticsearch-spark_2.10" % "2.3.3"
Comments welcome...
//Just showing the ES stuff
import org.elasticsearch.hadoop
import org.elasticsearch.spark._
//Declare a schema
val schemaString = "age:int, name:string,location:string"
//Fill RDD with dummy data
val rdd = sc.textFile("/path/to/your/file.csv")
val seperator = "," //This is seperator in csv
//Convert schema string above into a struct
val schema =
StructType(
schemaString.split(",").map(fieldName =>
StructField(fieldName.split(":")(0),
getFieldTypeInSchema(fieldName.split(":")(1)), true)))
//map each element of RDD row to RDD with elements
val rowRDDx =rdd.map(p => {
var list: collection.mutable.Seq[Any] = collection.mutable.Seq.empty[Any]
var index = 0
var tokens = p.split(seperator)
tokens.foreach(value => {
var valType = schema.fields(index).dataType
var returnVal: Any = null
valType match {
case IntegerType => returnVal = value.toString.toInt
case DoubleType => returnVal = value.toString.toDouble
case LongType => returnVal = value.toString.toLong
case FloatType => returnVal = value.toString.toFloat
case ByteType => returnVal = value.toString.toByte
case StringType => returnVal = value.toString
case TimestampType => returnVal = value.toString
}
list = list :+ returnVal
index += 1
})
Row.fromSeq(list)
})
//Convert the RDD with elements to a DF , also specify the intended schema
val df = sqlContext.createDataFrame(rowRDDx, schema)
//index the DF to ES
EsSparkSQL.saveToEs(df,"test/doc")