Convert dataframe to json in scala - mongodb

Question

Assuming i have a wordcount example where i get a dataframe as word in one column and wordcount in another column, I want to collect the same and store it as an array of json in mongo collection.
eg for dataframe:
|Word | Count |
| abc | 1 |
| xyz | 23 |
I should get the json like:
{words:[{word:"abc",count:1},{word:"xyz",count:23}]}
When i tried .toJSON on the dataframe and collected the value as list and added it to a dataframe the result which got stored in my mongo was a collection of string rather than collection of JSON.
query used :
explodedWords1.toJSON.toDF("words").agg(collect_list("words")).toDF("words")
result : "{\"words\":[{\"word\":\"abc\",\"count\":1},{\"word\":\"xyz\",\"count\":23}]}"
I am new to Scala. Any help will be good. (Will be helpful if external package aint used).

Answer 1

The absolute best way to store data from dataframes into Mongo is using the
MongoDB Spark Connector (https://docs.mongodb.com/spark-connector/master/)
Just add "org.mongodb.spark" %% "mongo-spark-connector" % "2.2.0" to your sbt dependencies and check code below
import com.mongodb.spark.MongoSpark
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder()
.master("local[2]")
.appName("test")
.config("spark.mongodb.input.uri", "mongodb://127.0.0.1/dbname")
.config("spark.mongodb.output.uri", "mongodb://127.0.0.1/dbname")
.getOrCreate()
import spark.implicits._
val explodedWords1 = List(
("abc",1),
("xyz",23)
).toDF("Word", "Count")
MongoSpark.save(explodedWords1.write.option("collection", "wordcount").mode("overwrite"))
However if you do want the results as a single json file then the script below should do it:
explodedWords1.repartition(1).write.json("/tmp/wordcount")
Finally if you want the json as a list of strings in your scala just use
explodedWords1.toJSON.collect()
Update:
I didn't see that you wanted all records aggregated to one field ("words")
If you use the code below, then all three methods above still function (swapping explodedWords1 with aggregated)
import org.apache.spark.sql.functions._
val aggregated = explodedWords1.agg(
collect_list(map(lit("word"), 'Word, lit("count"), 'Count)).as("words")
)
Option 1: explodedWords1
Option 2: aggregated