Error while executing aggregation query in Elastic search from Spark - scala

Question

I have written a function which takes the elastic query as input, executes on Elastic cluster and return back with the result.
When I pass simple query like this
val query = {
"query" = {
"match" : {
"healthSystemId.keyword":"XYZ"
}
}
}
it return back the executed query results.
But when I pass aggregation query like this
val query = {
"size": 0,
"aggregations": {
"corr_agg": {
"terms": {
"field": "healthSystemId.keyword",
"size": 5000
}
}
}
}
it throws the following error.
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 122.0 failed 4 times, most recent failure: Lost task 0.3 in stage 122.0 (TID 594) (10.139.64.24 executor 4): org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: org.elasticsearch.hadoop.rest.EsHadoopRemoteException: parsing_exception: [size] query malformed, no start_object after query name
{"query":{"size":0,"aggregations":{"corr_agg":{"terms":{"field":"healthSystemId.keyword","size":5000}}}},"_source":["healthSystemId"]}
at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:477)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:434)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:428)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:408)
at org.elasticsearch.hadoop.rest.RestRepository.scroll(RestRepository.java:311)
at org.elasticsearch.hadoop.rest.ScrollQuery.hasNext(ScrollQuery.java:93)
at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.hasNext(AbstractEsRDDIterator.scala:61)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:761)
at org.apache.spark.sql.execution.collect.UnsafeRowBatchUtils$.encodeUnsafeRows(UnsafeRowBatchUtils.scala:80)
at org.apache.spark.sql.execution.collect.Collector.$anonfun$processFunc$1(Collector.scala:186)
at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$3(ResultTask.scala:75)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$1(ResultTask.scala:75)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:55)
at org.apache.spark.scheduler.Task.doRunTask(Task.scala:169)
at org.apache.spark.scheduler.Task.$anonfun$run$4(Task.scala:137)
at com.databricks.unity.EmptyHandle$.runWithAndClose(UCSHandle.scala:104)
at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:137)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.scheduler.Task.run(Task.scala:96)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$13(Executor.scala:902)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1696)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:905)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:760)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
But when I execute this query in Elastic, it return me back with results. This is the helper method I am using to read from Elastic
def readFromElastic(mappingId:String,arrayFields:String,fieldsToInclude:String,index:String,query:String) = {
spark.read
.format("org.elasticsearch.spark.sql")
.option("es.port", <ES_PORT> )
.option("es.nodes", <ES_NODE> )
.option("es.nodes.wan.only", <ES_WAN_ONLY> )
.option("es.net.ssl", sys.env("ES_NET_SSL"))
.option("es.net.http.auth.user", <ES_USER> )
.option("es.net.http.auth.pass", <ES_PASSWORD> )
.option("es.field.read.empty.as.null", "no")
.option("es.mapping.id",mappingId)
.option("es.read.field.as.array.include",arrayFields)
.option("es.read.field.include", fieldsToInclude)
.option("pushdown", "true")
.option("es.query", query )
.load(index)
}
Any idea on what possibly could be wrong here