How to save a JSON file in ElasticSearch using Spark? - scala

Question

I am trying to save a JSON file in ElasticSearch but its not working.
This is my code:
import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
import org.elasticsearch.spark.sql._
import org.apache.spark.SparkConf
object HelloEs {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setAppName("WriteToES").setMaster("local")
conf.set("es.index.auto.create", "true")
val sc = new SparkContext(conf)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val sen_p = sqlContext.read.json("/home/Bureau/mydoc/Orange.json")
sen_p.registerTempTable("sensor_ptable")
sen_p.saveToEs("sensor/metrics")
}
}
I am getting also this error:
Exception in thread "main" java.lang.NoSuchMethodError: org.elasticsearch.spark.sql.package$.sparkDataFrameFunctions(Lorg/apache/spark/sql/Dataset;)Lorg/elasticsearch/spark/sql/package$SparkDataFrameFunctions;
at learnscala.HelloEs$.main(HelloEs.scala:20)
at learnscala.HelloEs.main(HelloEs.scala)

Answer 1

There are multiple ways to save an RDD / Dataframe to Elastic Search.
Spark Dataframe can be written to Elastic Search using:
df.write.format("org.elasticsearch.spark.sql").mode("append").option("es.resource","<ES_RESOURCE_PATH>").option("es.nodes", "http://<ES_HOST>:9200").save()
RDD can be written to ES using:
import org.elasticsearch.spark.rdd.EsSpark
EsSpark.saveToEs(rdd, "<ES_RESOURCE_PATH>")
In your case, modify the code as below:
`
object HelloEs {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setAppName("WriteToES").setMaster("local")
conf.set("es.index.auto.create", "true")
val sc = new SparkContext(conf)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val sen_p = sqlContext.read.json("/home/Bureau/mydoc/Orange.json")
sen_p.write.format("org.elasticsearch.spark.sql").mode("append").option("es.resource","<ES_RESOURCE_PATH>").option("es.nodes", "http://<ES_HOST>:9200").save()
}
}
`