I'm trying to write a function that aggregates data from my MongoDB using ReactiveMongo (0.12), with Play JSON serialisation (similar to this question).
So this is what I have:
def getPopAggregate(col: JSONCollection) = {
import col.BatchCommands.AggregationFramework.{AggregationResult, Group, Match, SumField}
col.aggregate(
Group(JsString("$rstId"))("totalPopulation" -> SumField("population")),
List(Match(Json.obj("totalPopulation" -> Json.obj("$gte" -> 1000))))
).map(_.firstBatch)
}
This outputs Future[List[JsObject]], however I want to map the results to a List of my case class (i.e. Future[Seq[PopAggregate]]).
case class PopAggregate(rstId: Option[BSONObjectID], totalPopulation: Double)
object PopAggregate {
implicit val popAggregateFormat = Json.format[PopAggregate]
}
I hope someone can spare a moment to help me past this one. Many thanks!
Related
I am new to Scala and was trying my hands on with akka. I am trying to access data from MongoDB in Scala and want to convert it into JSON and XML format.
This code attached below is using path /getJson and calling getJson() function to get data in a form of future.
get {
concat(
path("getJson"){
val f = Patterns.ask(actor1,getJson(),10.seconds)
val res = Await.result(f,10.seconds)
val result = res.toString
complete(res.toString)
}
}
The getJson() method is as follows:
def getJson()= {
val future = collection.find().toFuture()
future
}
I have a Greeting Case class in file Greeting.scala:
case class Greeting(msg:String,name:String)
And MyJsonProtocol.scala file for Marshelling of scala object to JSON format as follows:
trait MyJsonProtocol extends SprayJsonSupport with DefaultJsonProtocol {
implicit val templateFormat = jsonFormat2(Greeting)
}
I am getting output of complete(res.toString) in Postman as :
Future(Success(List(
Iterable(
(_id,BsonObjectId{value=5fc73944986ced2b9c2527c4}),
(msg,BsonString{value='Hiiiiii'}),
(name,BsonString{value='Ruchirrrr'})
),
Iterable(
(_id,BsonObjectId{value=5fc73c35050ec6430ec4b211}),
(msg,BsonString{value='Holaaa Amigo'}),
(name,BsonString{value='Pablo'})),
Iterable(
(_id,BsonObjectId{value=5fc8c224e529b228916da59d}),
(msg,BsonString{value='Demo'}),
(name,BsonString{value='RuchirD'}))
)))
Can someone please tell me how to iterate over this output and to display it in JSON format?
When working with Scala, its very important to know your way around types. First step toweards this is at least knowing the types of your variables and values.
If you look at this method,
def getJson() = {
val future = collection.find().toFuture()
future
}
Is lacks the type type information at all levels, which is a really bad practice.
I am assuming that you are using mongo-scala-driver. And your collection is actually a MongoCollection[Document].
Which means that the output of collection.find() should be a FindOberservable[Document], hence collection.find().toFuture() should be a Future[Seq[Document]]. So, your getJson method should be written as,
def getJson(): Future[Seq[Document]] =
collection.find().toFuture()
Now, this means that you are passing a Future[Seq[Document]] to your actor1, which is again a bad practice. You should never send any kind of Future values among actors. It looks like your actor1 does nothing but sends the same message back. Why does this actor1 even required when it does nothing ?
Which means your f is a Future[Future[Seq[Document]]]. Then you are using Await.result to get the result of this future f. Which is again an anti-pattern, since Await blocks your thread.
Now, your res is a Future[Seq[Document]]. And you are converting it to a String and sending that string back with complete.
Your JsonProtocol is not working because you are not even passing it any Greeting's.
You have to do the following,
Read raw Bson objects from mongo.
convert raw Bson objects to your Gretting objects.
comlete your result with these Gretting objects. The JsonProtocol should take case of converting these Greeting objects to Json.
The easist way to do all this is by using the mongo driver's CodecRegistreis.
case class Greeting(msg:String, name:String)
Now, your MongoDAL object will look like following (it might be missing some imports, fill any missing imports as you did in your own code).
import org.mongodb.scala.bson.codecs.Macros
import org.mongodb.scala.bson.codecs.DEFAULT_CODEC_REGISTRY
import org.bson.codecs.configuration.CodecRegistries
import org.mongodb.scala.{MongoClient, MongoCollection, MongoDatabase}
object MongoDAL {
val greetingCodecProvider = Macros.createCodecProvider[Greeting]()
val codecRegistry = CodecRegistries.fromRegistries(
CodecRegistries.fromProviders(greetingCodecProvider),
DEFAULT_CODEC_REGISTRY
)
val mongoClient: MongoClient = ... // however you are connecting to mongo and creating a mongo client
val mongoDatabase: MongoDatabase =
mongoClient
.getDatabase("database_name")
.withCodecRegistry(codecRegistry)
val greetingCollection: MongoCollection[Greeting] =
mongoDatabase.getCollection[Greeting]("greeting_collection_name")
def fetchAllGreetings(): Future[Seq[Greeting]] =
greetingCollection.find().toFuture()
}
Now, your route can be defined as
get {
concat(
path("getJson") {
val greetingSeqFuture: Future[Seq[Greeting]] = MongoDAL.fetchAllGreetings()
// I don't see any need for that actor thing,
// but if you really need to do that, then you can
// do that by using flatMap to chain future computations.
val actorResponseFuture: Future[Seq[Greeting]] =
greetingSeqFuture
.flatMap(greetingSeq => Patterns.ask(actor1, greetingSeq, 10.seconds))
// complete can handle futures just fine
// it will wait for futre completion
// then convert the seq of Greetings to Json using your JsonProtocol
complete(actorResponseFuture)
}
}
First of all, don't call toString in complete(res.toString).
As it said in AkkaHTTP json support guide if you set everything right, your case class will be converted to json automatically.
But as I see in the output, your res is not an object of a Greeting type. Looks like it is somehow related to the Greeting and has the same structure. Seems to be a raw output of the MongoDB request. If it is a correct assumption, you should convert the raw output from MongoDB to your Greeting case class.
I guess it could be done in getJson() after collection.find().
I have a case class with simple data:
case class MyClass(
details: Details,
names: List[String],
id: String,
)
I have created a couchbase query which should retrieve all documents from database:
val query = s"SELECT * from `docs`"
for {
docs<- bucket
.query(N1qlQuery.simple(query))
.flatMap((rows: AsyncN1qlQueryResult) => rows.rows())
.toList
.parse[F]
.map(_.asScala.toList)
} yield docs
parse[F] is a simple function to convert from Observable. The problem here is that I got an error type mismatch which says that found List[AsyncN1qlQueryResult] instead of required List[MyClass]. How should I convert from AsyncN1qlQueryResult into MyClass objects?
I'm using Circe to parse documents.
I'm happy to report that there is now an early release of the native Couchbase Scala SDK available, which does include support for converting each row result of a N1QL query directly into your case class:
case class Address(line1: String)
case class User(name: String, age: Int, addresses: Seq[Address])
object User {
// Define a Codec so SDK knows how to convert User to/from JSON
implicit val codec: Codec[User] = Codecs.codec[User]
}
val statement = """select * from `users`;"""
val rows: Try[Seq[User]] = cluster.query(statement)
.map(result => result
.rows.flatMap(row =>
row.contentAs[User].toOption))
rows match {
case Success(rows: Seq[User]) =>
rows.foreach(row => println(row))
case Failure(err) =>
println(s"Error: $err")
}
This is the blocking API. There's also APIs to allow getting the results as Futures, or as Flux/Monos from Reactive Programming, so you have a lot of flexibility on how to get the data.
You can see how to get started here: https://docs.couchbase.com/scala-sdk/1.0alpha/hello-world/start-using-sdk.html
Please note that this is an alpha release to let the community get an idea where we're heading with it and give them an opportunity to provide feedback. It shouldn't be used in production. The forums (https://forums.couchbase.com/) are the best place to raise any feedback you have.
I am trying to prevent empty values being inserted into my mongoDB collection. The field in question looks like this:
MongoDB Field
"stadiumArr" : [
"Old Trafford",
"El Calderon",
...
]
Sample of (mapped) case class
case class FormData(_id: Option[BSONObjectID], stadiumArr: Option[List[String]], ..)
Sample of Scala form
object MyForm {
val form = Form(
mapping(
"_id" -> ignored(Option.empty[BSONObjectID]),
"stadiumArr" -> optional(list(text)),
...
)(FormData.apply)(FormData.unapply)
)
}
I am also using the Repeated Values functionality in Play Framework like so:
Play Template
#import helper._
#(myForm: Form[models.db.FormData])(implicit request: RequestHeader, messagesProvider: MessagesProvider)
#repeatWithIndex(myForm("stadiumArr"), min = 5) { (stadium, idx) =>
#inputText(stadium, '_label -> ("stadium #" + (idx + 1)))
}
This ensures that whether there are at least 5 values or not in the array; there will still be (at least) 5 input boxes created. However if one (or more) of the input boxes are empty when the form is submitted an empty string is still being added as value in the array, e.g.
"stadiumArr" : [
"Old Trafford",
"El Calderon",
"",
"",
""
]
Based on some other ways of converting types from/to the database; I've tried playing around with a few solutions; such as:
implicit val arrayWrite: Writes[List[String]] = new Writes[List[String]] {
def writes(list: List[String]): JsValue = Json.arr(list.filterNot(_.isEmpty))
}
.. but this isn't working. Any ideas on how to prevent empty values being inserted into the database collection?
Without knowing specific versions or libraries you're using it's hard to give you an answer, but since you linked to play 2.6 documentation I'll assume that's what you're using there. The other assumption I'm going to make is that you're using reactive-mongo library. Whether or not you're using the play plugin for that library or not is the reason why I'm giving you two different answers here:
In that library, with no plugin, you'll have defined a BSONDocumentReader and a BSONDocumentWriter for your case class. This might be auto-generated for you with macros or not, but regardless how you get it, these two classes have useful methods you can use to transform the reads/writes you have to another one. So, let's say I defined a reader and writer for you like this:
import reactivemongo.bson._
case class FormData(_id: Option[BSONObjectID], stadiumArr: Option[List[String]])
implicit val formDataReaderWriter = new BSONDocumentReader[FormData] with BSONDocumentWriter[FormData] {
def read(bson: BSONDocument): FormData = {
FormData(
_id = bson.getAs[BSONObjectID]("_id"),
stadiumArr = bson.getAs[List[String]]("stadiumArr").map(_.filterNot(_.isEmpty))
)
}
def write(formData: FormData) = {
BSONDocument(
"_id" -> formData._id,
"stadiumArr" -> formData.stadiumArr
)
}
}
Great you say, that works! You can see in the reads I went ahead and filtered out any empty strings. So even if it's in the data, it can be cleaned up. That's nice and all, but let's notice I didn't do the same for the writes. I did that so I can show you how to use a useful method called afterWrite. So pretend the reader/writer weren't the same class and were separate, then I can do this:
val initialWriter = new BSONDocumentWriter[FormData] {
def write(formData: FormData) = {
BSONDocument(
"_id" -> formData._id,
"stadiumArr" -> formData.stadiumArr
)
}
}
implicit val cleanWriter = initialWriter.afterWrite { bsonDocument =>
val fixedField = bsonDocument.getAs[List[String]]("stadiumArr").map(_.filterNot(_.isEmpty))
bsonDocument.remove("stadiumArr") ++ BSONDocument("stadiumArr" -> fixedField)
}
Note that cleanWriter is the implicit one, that means when the insert call on the collection happens, it will be the one chosen to be used.
Now, that's all a bunch of work, if you're using the plugin/module for play that lets you use JSONCollections then you can get by with just defining play json Reads and Writes. If you look at the documentation you'll see that the reads trait has a useful map function you can use to transform one Reads into another.
So, you'd have:
val jsonReads = Json.reads[FormData]
implicit val cleanReads = jsonReads.map(formData => formData.copy(stadiumArr = formData.stadiumArr.map(_.filterNot(_.isEmpty))))
And again, because only the clean Reads is implicit, the collection methods for mongo will use that.
NOW, all of that said, doing this at the database level is one thing, but really, I personally think you should be dealing with this at your Form level.
val form = Form(
mapping(
"_id" -> ignored(Option.empty[BSONObjectID]),
"stadiumArr" -> optional(list(text)),
...
)(FormData.apply)(FormData.unapply)
)
Mainly because, surprise surprise, form has a way to deal with this. Specifically, the mapping class itself. If you look there you'll find a transform method you can use to filter out empty values easily. Just call it on the mapping you need to modify, for example:
"stadiumArr" -> optional(
list(text).transform(l => l.filter(_.nonEmpty), l => l.filter(_.nonEmpty))
)
To explain a little more about this method, in case you're not used to reading the signatures in the scaladoc.
def
transform[B](f1: (T) ⇒ B, f2: (B) ⇒ T): Mapping[B]
says that by calling transform on some mapping of type Mapping[T] you can create a new mapping of type Mapping[B]. In order to do this you must provide functions that convert from one to the other. So the code above causes the list mapping (Mapping[List[String]]) to become a Mapping[List[String]] (the type did not change here), but when it does so it removes any empty elements. If I break this code down a little it might be more clear:
def convertFromTtoB(list: List[String]): List[String] = list.filter(_.nonEmpty)
def convertFromBtoT(list: List[String]): List[String] = list.filter(_.nonEmpty)
...
list(text).transform(convertFromTtoB, convertFromBtoT)
You might wondering why you need to provide both, the reason is because when you call Form.fill and the form is populated with values, the second method will be called so that the data goes into the format the play form is expecting. This is more obvious if the type actually changes. For example, if you had a text area where people could enter CSV but you wanted to map it to a form model that had a proper List[String] you might do something like:
def convertFromTtoB(raw: String): List[String] = raw.split(",").filter(_.nonEmpty)
def convertFromBtoT(list: List[String]): String = list.mkString(",")
...
text.transform(convertFromTtoB, convertFromBtoT)
Note that when I've done this in the past sometimes I've had to write a separate method and just pass it in if I didn't want to fully specify all the types, but you should be able to work from here given the documentation and type signature for the transform method on mapping.
The reason I suggest doing this in the form binding is because the form/controller should be the one with the concern of dealing with your user data and cleaning things up I think. But you can always have multiple layers of cleaning and whatnot, it's not bad to be safe!
I've gone for this (which always seems obvious when it's written and tested):
implicit val arrayWrite: Writes[List[String]] = new Writes[List[String]] {
def writes(list: List[String]): JsValue = Json.toJson(list.filterNot(_.isEmpty).toIndexedSeq)
}
But I would be interested to know how to
.map the existing Reads rather than redefining from scratch
as #cchantep suggests
My database looks like
[
{
name: "domenic",
records: {
today: 5,
yesterday: 1.5
}
},
{
name: "bob",
records: { ... }
}
]
When I try queries like
val result: Option[DBObject] = myCollection.findOne(
MongoDBObject("name" -> "domenic")
MongoDBObject("records" -> 1),
)
val records = result.get.getAs[BasicDBObject]("records").get
grater[Map[String, Number]].asObject(records)
it fails (at runtime!) with
GRATER GLITCH - unable to find or instantiate a grater using supplied path name
REASON: Class scala.collection.immutable.Map is an interface
Context: 'global'
Path from pickled Scala sig: 'scala.collection.immutable.Map'
I think I could make this work by creating a case class whose only field is a Map[String, Number] and then getting its property. Is that really necessary?
grater doesn't take a collection as a type argument, only a case class or a trait/abstract class whose concrete representations are case classes. Since you're just querying for a map, just extract the values you need out of the DBObject using getAs[T].
Number may not be a supported type in Salat - I've certainly never tried it. If you need Number you can write a custom transformer or send a pull request to add real support to Salat.
HI..
I am working with the scala n mongodb.
now i want access mongodb database in scala swing application.
so which drivers can i use for it?
and which can easily work?
please reply
I've been using casbah http://api.mongodb.org/scala/casbah/2.0.2/index.html to talk to mongodb from my scala swing application.
It's pretty easy to install and setup, and the API is quite scala-esque.
The hardest part is understanding mongodb itself, (coming from an sql background)
We were sort of unsatisfied with the way Casbah works for deep objects or simple maps and no real case class mapping support so we rolled our own MongoDB Synchronous Scala driver on top of the legacy java driver which I would like to shamelessly plug here with an example on how to store and retrieve a map and a simple case class. The driver does not have a lot of magic and is easy to setup and features a simple BSON implementation which was inspired by the Play2 JSON impl.
Here is how to use it with some simple values:
val client = MongoClient("hostname", 27017)
val db = client("dbname")
val coll = db("collectionname")
coll.save(Bson.doc("_id" -> 1, "vals" -> Map("key1" -> "val1")))
val docOpt = coll.findOneById(1) // => Option[BsonDoc]
for(doc <- docOpt)
println(doc.as[Map[String, String]]("vals")("key1")) // => prints "val1"
For a case class it is a little bit more complex but it is all handrolled and there is no magic involved so you can do whatever you like and how you need it, i.e. provider some shorter key names in the doc:
case class DnsRecord(host: String = "", ttl: Long = 0, otherProps: Map[String, String] = Map())
case object DnsRecord {
implicit object DnsRecordToBsonElement extends ToBsonElement[DnsRecord] {
def toBson(v: DnsRecord): BsonElement = DnsRecordToBsonDoc.toBson(v)
}
implicit object DnsRecordFromBsonElement extends FromBsonElement[DnsRecord] {
def fromBson(v: BsonElement): DnsRecord = DnsRecordFromBsonDoc.fromBson(v.asInstanceOf[BsonDoc])
}
implicit object DnsRecordFromBsonDoc extends FromBsonDoc[DnsRecord] {
def fromBson(d: BsonDoc): DnsRecord = DnsRecord(
d[String]("host"),
d[Long]("ttl"),
d[Map[String, String]]("op")
)
}
implicit object DnsRecordToBsonDoc extends ToBsonDoc[DnsRecord] {
def toBson(m: DnsRecord): BsonDoc = Bson.doc(
"host" -> m.host,
"ttl" -> m.ttl,
"op" -> m.otherProps
)
}
}
coll.save(DnsRecord("test.de", 4456, Map("p2" -> "val1")))
for (r <- coll.findAs[DnsRecord](Bson.doc("host" -> "test.de")))
println(r.host)
As an update for people finding this thread and interested in MongoDB 3.X. We're using Async driver which can be found here https://github.com/evojam/mongodb-driver-scala. It's API is build in Scala way with new Play 2.4 module ready if you're using it, but you can always take only driver.