How to connect to ElasticSearch from Apache Spark Service on Bluemix - scala

Question

I use Apache Spark service on Bluemix to create demo (collecting/parsing twitter data).I want to transport Elastic Search.
I created my scala app according to the following URL [1]:
[1] https://www.elastic.co/guide/en/elasticsearch/hadoop/current/spark.html
However, when using Jupyter notebook on Bluemix, I couldn't run my app properly. A special interpreter-aware SparkContext "sc" was already running, but I cloudn't add properties to "sc" such as "es.nodes", "es.port" ,and so on to connect Elastic Search.
Q1.
Does anyone know how to add extra properties to a special interpreter-aware SparkContext on Bluemix? In my local spark environment, it's easy to add.
Q2.
I tried to create another SparkContext as follows and use for streaming, but it was uncontrollable on Jupyter notebook..
var conf = sc.getConf
conf.set("es.index.auto.create", "true")
conf.set("es.nodes", "XXXXXXXX")
conf.set("es.port", "9020")
conf.set("spark.driver.allowMultipleContexts", "true")
val sc1 = new SparkContext(conf)
My procedure to create extra SparkContext may not be right, I think.
Does anyone know how to create 2nd SparkContext properly on Bluemix?

Answer 1

If I'm not mistaken, you're already setting the properties on the configuration object within the existing SparkContext.
These lines (correcting what I assume is a typo) should be setting the option on the existing SparkContext's configuration:
val conf = sc.getConf
conf.set("es.index.auto.create", "true")
conf.set("es.nodes", "XXXXXXXX")
conf.set("es.port", "9020")
conf.set("spark.driver.allowMultipleContexts", "true")
You mentioned you couldn't add these properties -- can you elaborate on the problem it was causing doing it this way?