Read Json files over List of HTTPs location using Spark - scala

Question

I'm trying to read JSON files from HTTP using spark. Since it's not an HDFS or any place where Spark can read data easily and convert it to a data frame. The URL(S) is HTTPS and needs a token and a bunch of headers to successfully retrieve the response. Is there a way to achieve this task? The response is like this which can be easily converted to a row in the data frame.
{
"code": "403010",
"message": "message 1"
}
{
"code": "403010",
"message": "message 1"
}
{
"code": "403010",
"message": "message 1"
}
Now the response is weird because there are multiple JSON heads but it is the actual response from the API.

Answer 1

answer is provided to this url.(for others) Get Results From URL using Scala Spark
import org.apache.spark.sql.{DataFrame, SQLContext, SparkSession}
def GetUrlContentJson(url: String): DataFrame ={
val result = scala.io.Source.fromURL(url).mkString
//only one line inputs are accepted. (I tested it with a complex Json and it worked)
val jsonResponseOneLine = result.toString().stripLineEnd
//You need an RDD to read it with spark.read.json! This took me some time. However it seems obvious now
val jsonRdd = spark.sparkContext.parallelize(jsonResponseOneLine :: Nil)
val jsonDf = spark.read.json(jsonRdd)
return jsonDf
}
val response = GetUrlContentJson(url)
response.show