Dataframe becomes empty after appending an dataframe inside scala for loop - scala

Question

Actually, I am trying to append a dataframe to an empty dataframe in for loop in scala.
but the appended dataframe becomes empty every time.
below is the code
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql._
import org.apache.spark.sql.types._
import org.apache.spark.sql.functions._
import java.io._
import org.apache.spark.sql.DataFrame
object obj_Spark_url_Zipcode {
def main(args:Array[String]):Unit={
val spark = SparkSession.builder().appName("Spark_Url_Zip").master("local[*]").getOrCreate()
val sc = spark.sparkContext
import spark.implicits._
sc.setLogLevel("Error")
System.setProperty("http.agent","chrome")
val schema_str = "first,gender,state,zip,phone"
val struct_dymic = StructType(schema_str.split(",").map(x=>StructField(x, StringType, true)))
val df_empty = spark.createDataFrame(sc.emptyRDD[Row],struct_dymic)
for(i <- 1 to 10)
{
val url_json_data = scala.io.Source.fromURL("https://webapiusr.mue/apii/0.05/?reslts=4554").mkString
val url_json_rdd = sc.parallelize(url_json_data::Nil) //To convert a string to RDD
val url_json_df = spark.read.option("multiline",true).json(url_json_rdd)
val zipcode_df = url_json_df.withColumn("results",explode(col("results")))
.select("results.user.name.first","results.user.gender","results.user.location.state","results.user.location.zip","results.user.phone")
df_empty.union(zipcode_df)
println("Curr val : "+i)
}
df_empty.show()
}
}
Result:
#######
Curr val : 1
Curr val : 2
Curr val : 3
Curr val : 4
Curr val : 5
Curr val : 6
Curr val : 7
Curr val : 8
Curr val : 9
Curr val : 10
+-----+------+-----+---+-----+
|first|gender|state|zip|phone|
+-----+------+-----+---+-----+
+-----+------+-----+---+-----+
my intention is to append all the dataframes created inside the for loop into one dataframe and write the final dataframe into target.
I don't know why it becomes empty.
I tried this approach in pyspark. Appending the dataframes into an array and union array of dataframes into one dataframe.
But in scala, am unable to add dataframes into an array. (array of dataframes)
Regards
Dinesh Kumar

Answer 1

Example in Scala
import spark.implicits._
case class ReduceUnion (id: Int, v: String)
val l = Array.range(1,10)
val d = l.map(i => Seq(ReduceUnion(i, s"Test $i")).toDF())
val resultDF = d.reduce(_ union _)
resultDF.printSchema()
resultDF.show(false)
// root
// |-- id: integer (nullable = false)
// |-- v: string (nullable = true)
//
// +---+------+
// |id |v |
// +---+------+
// |1 |Test 1|
// |2 |Test 2|
// |3 |Test 3|
// |4 |Test 4|
// |5 |Test 5|
// |6 |Test 6|
// |7 |Test 7|
// |8 |Test 8|
// |9 |Test 9|
// +---+------+