How do I compose nested case classes populated with async mongodb queries in play framework - scala

I have been trying to convert mongodb queries I have working with await to using totally async. The problem is, I cannot find any examples or get code working to populate a list of objects where for each object there is a nested find returning futures.
I have seen examples for a single object, such as
val user = mongoDao.getUser(id)
val address = mongoDao.getAddress(user.id)
for that I see for comprehension works just fine. However, I have a list of objects (similar to users) and I cant seem to get the code right.
What I need to do is get all the users in an async manner, then when they complete, get all the addresses and populate a field (or create a new case class.)
val usersFuture : Future[List[User]] = mongoDao.getUsers()
val fullFutures : Future[List[FullUser]] = usersFuture.map(users: List[User] => {
users.map(user: User => {
val futureAddress : Future[Address] = mongoDao.getAddress()
// Now create a object
futureAddress.map(address: Address) {
FullUserInfo(user, address)
}
}
}
So, I'd like to end up with a Future[List[FullUser]] that I can return to the play framework. I've included the cutdown I've tried.
thanks
// OBJECTS HERE
case class Outer(id: Int, name: String)
case class Inner(id: Int, name: String)
case class Combined(id: Int, name: String, inner: Inner)
// FAKE DAO to reproduct
#Singleton
class StatInner #Inject()( implicit val ec: ExecutionContext) {
def outer() = {
Future {
val lb = new ListBuffer[Outer]()
Thread.sleep(1000)
println("Done")
for (id <- 1 to 5) {
lb += Outer(id, s"Hello $id")
}
lb.toList
}
}
def inner(id: Int) : Future[Inner] = {
Future {
Thread.sleep(1000)
Inner(id, s"inner $id")
}
}
}
// CODE to query that is not working
def nestedTree = Action.async {
val statInner : StatInner = new StatInner()
val listouter : Future[List[Outer]] = statInner.outer()
val combined = listouter.map((listOuter : List[Outer]) => {
listOuter.flatMap((outer: Outer) => {
val futInner : Future[Inner] = statInner.inner(outer.id)
futInner.map((inner: Inner) => {
Combined(outer, inner)
})
})
})
combined.map(Json.toJson(_))
}
```

Use Future.{flatMap.sequence}:
val usersFuture: Future[List[User]] = mongoDao.getUsers()
val fullFutures: Future[List[FullUser]] = usersFuture.flatMap { users =>
Future.sequence(users.map { user =>
mongoDao.getAddress().map { adress =>
FullUserInfo(user, address)
}
})
}

Related

Comparing the json data types at runtime using Jackson and Scala

I have an incoming JSON data that looks like below:
{"id":"1000","premium":29999,"eventTime":"2021-12-22 00:00:00"}
Now, I have created a class that will accept this record and will check whether the data type of the incoming record is according to the data types defined in the case class. However, when I am calling the method it is always calling the Failure part of the match case.
case class Premium(id: String, premium: Long, eventTime: String)
class Splitter extends ProcessFunction[String, Premium] {
val outputTag = new OutputTag[String]("failed")
def fromJson[T](json: String)(implicit m: Manifest[T]): Either[String, T] = {
Try {
println("inside")
lazy val mapper = new ObjectMapper() with ScalaObjectMapper
mapper.registerModule(DefaultScalaModule)
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
mapper.readValue[T](json)
} match {
case Success(x) => {
Right(x)
}
case Failure(err) => {
Left(json)
}
}
}
override def processElement(i: String, context: ProcessFunction[String, Premium]#Context, collector: Collector[Premium]): Unit = {
fromJson(i) match {
case Right(data) => {
collector.collect(data)
println("Good Records: " + data)
}
case Left(json) => {
context.output(outputTag, json)
println("Bad Records: " + json)
}
}
}
}
Based on the sample record above, it should pass the Success value but no matter what I pass, it always enters the Failure part. What else is missing?
I am using Scala 2.11.12 and I tried examples from this link and this link but no luck.

How to integrate akka streams kafka (reactive-kafka) into akka http application?

I have a basic scala akka http CRUD application. See below for the relevant classes.
I'd simply like to write an entity id and some data (as json) to a Kafka topic whenever, for example, an entity is created/updated.
I'm looking at http://doc.akka.io/docs/akka-stream-kafka/current/producer.html, but am new to scala and akka, and unsure of how to integrate it into my application?
For example, from the docs above, this is the example of a producer writing to kafka, so I think I need to something similar, but whereabouts in my application should this go? Can I just add another map call in the create method in my service after I have created the user?
Many thanks!
val done = Source(1 to 100)
.map(_.toString)
.map { elem =>
new ProducerRecord[Array[Byte], String]("topic1", elem)
}
.runWith(Producer.plainSink(producerSettings))
Or do I need to do something like the example here https://github.com/hseeberger/accessus in the bindAndHandle() method in my Server.scala?
WebServer.scala
object System {
implicit val system = ActorSystem()
implicit val dispatcher = system.dispatcher
implicit val actorMaterializer = ActorMaterializer()
}
object WebServer extends App {
import System._
val config = new ApplicationConfig() with ConfigLoader
ConfigurationFactory.setConfigurationFactory(new LoggingConfFileConfigurationFactory(config.loggingConfig))
val injector = Guice.createInjector(new MyAppModule(config))
val routes = injector.getInstance(classOf[Router]).routes
Http().bindAndHandle(routes, config.httpConfig.interface, config.httpConfig.port)
}
Router.scala
def routes(): Route = {
post {
entity(as[User]) { user =>
val createUser = userService.create(user)
onSuccess(createUser) {
case Invalid(y: NonEmptyList[Err]) => {
throw new ValidationException(y)
}
case Valid(u: User) => {
complete(ToResponseMarshallable((StatusCodes.Created, u)))
}
}
}
} ~
// More routes here, left out for example
}
Service.scala
def create(user: User): Future[MaybeValid[User]] = {
for {
validating <- userValidation.validateCreate(user)
result <- validating match {
case Valid(x: User) =>
userRepo.create(x)
.map(dbUser => Valid(UserConverters.fromUserRow(x)))
case y: DefInvalid =>
Future{y}
}
} yield result
}
Repo.scala
def create(user: User): Future[User] = {
mutateDbProvider.db.run(
userTable returning userTable.map(_.userId)
into ((user, id) => user.copy(userId = id)) +=
user.copy(createdDate = Some(Timestamp.valueOf(LocalDateTime.now())))
)
}
Since you have written your Route to unmarshall just 1 User from the Entity I don't think you need Producer.plainSink. Rather, I think Producer.send will work just as well. Also, as a side note, throwing exceptions is not "idiomatic" scala. So I changed the logic for invalid user:
val producer : KafkaProducer = new KafkaProducer(producerSettings)
val routes : Route =
post {
entity(as[User]) { user =>
val createUser = userService.create(user)
onSuccess(createUser) {
case Invalid(y: NonEmptyList[Err]) =>
complete(BadRequest -> "invalid user")
case Valid(u: User) => {
val producerRecord =
new ProducerRecord[Array[Byte], String]("topic1",s"""{"userId" : ${u.userId}, "entity" : "User"}""")
onComplete(producer send producerRecord) { _ =>
complete(ToResponseMarshallable((StatusCodes.Created, u)))
}
}
}
}
}

scala returns doesn't conform to required S_

I got the error
found : scala.concurrent.Future[Option[models.ProcessTemplatesModel]]
required: Option[models.ProcessTemplatesModel]
My function is below
def createCopyOfProcessTemplate(processTemplateId: Int): Future[Option[ProcessTemplatesModel]] = {
val action = processTemplates.filter(_.id === processTemplateId).result.map(_.headOption)
val result: Future[Option[ProcessTemplatesModel]] = db.run(action)
result.map { case (result) =>
result match {
case Some(r) => {
var copy = (processTemplates returning processTemplates.map(_.id)) += ProcessTemplatesModel(None, "[Copy of] " + r.title, r.version, r.createdat, r.updatedat, r.deadline, r.status, r.comment, Some(false), r.checkedat, Some(false), r.approvedat, false, r.approveprocess, r.trainingsprocess)
val composedAction = copy.flatMap { id =>
processTemplates.filter(_.id === id).result.headOption
}
db.run(composedAction)
}
}
}
}
what is my problem in this case?
edit:
my controller function looks like this:
def createCopyOfProcessTemplate(processTemplateId: Int) = Action.async {
processTemplateDTO.createCopyOfProcessTemplate(processTemplateId).map { process =>
Ok(Json.toJson(process))
}
}
Is there my failure?
According to the your code - there are the following issues:
You use two db.run which return futures, but inner future will
not complete. For resolving it you should compose futures with
flatMap or for-comprehension.
You use only one partial-function case Some(_) => for pattern matching
and don't handle another value None.
You can use only one db.run and actions composition.
Your code can be like as:
def createCopyOfProcessTemplate(processTemplateId: Int): Future[Option[ProcessTemplatesModel]] = {
val action = processTemplates.filter(...).result.map(_.headOption)
val composedAction = action.flatMap {
case Some(r) =>
val copyAction = (processTemplates returning processTemplates...)
copyAction.flatMap { id =>
processTemplates.filter(_.id === id).result.headOption
}
case _ =>
DBIO.successful(None) // issue #2 has been resolved here
}
db.run(composedAction) // issue #3 has been resolved here
}
We get rid of issue #1 (because we use actions composition).

ReactiveMongo query returning None

I am just new to learning Scala and the related technologies.I am coming across the problem where the loadUser should return a record but its coming empty.
I am getting the following error:
java.util.NoSuchElementException: None.get
I appreciate this is not ideal Scala, so feel free to suggest me improvements.
class MongoDataAccess extends Actor {
val message = "Hello message"
override def receive: Receive = {
case data: Payload => {
val user: Future[Option[User]] = MongoDataAccess.loadUser(data.deviceId)
val twillioApiAccess = context.actorOf(Props[TwillioApiAccess], "TwillioApiAccess")
user onComplete {
case Failure(exception) => println(exception)
case p: Try[Option[User]] => p match {
case Failure(exception) => println(exception)
case u: Try[Option[User]] => twillioApiAccess ! Action(data, u.get.get.phoneNumber, message)
}
}
}
case _ => println("received unknown message")
}
}
object MongoDataAccess extends MongoDataApi {
def connect(): Future[DefaultDB] = {
// gets an instance of the driver
val driver = new MongoDriver
val connection = driver.connection(List("192.168.99.100:32768"))
// Gets a reference to the database "sensor"
connection.database("sensor")
}
def props = Props(new MongoDataAccess)
def loadUser(deviceId: UUID): Future[Option[User]] = {
println(s"Loading user from the database with device id: $deviceId")
val query = BSONDocument("deviceId" -> deviceId.toString)
// By default, you get a Future[BSONCollection].
val collection: Future[BSONCollection] = connect().map(_.collection("profile"))
collection flatMap { x => x.find(query).one[User] }
}
}
Thanks
There is no guaranty the find-one (.one[T]) matches at least one document in your DB, so you get an Option[T].
Then it's up to you to consider (or not) that having found no document is a failure (or not); e.g.
val u: Future[User] = x.find(query).one[User].flatMap[User] {
case Some(matchingUser) => Future.successful(matchingUser)
case _ => Future.failed(new MySemanticException("No matching user found"))
}
Using .get on Option is a bad idea anyway.

Dynamic orderBy with Squeryl

I can not figure out how to change orderBy dynamically in runtime. I need something like:
def samplesSorted(fields: List[String]) = {
from(Schema.samples)(s => select(s) orderBy(fields.map(getterByName))
}
Or something like
def samplesSorted(fields: List[String]) = {
val q = from(Schema.samples)(s => select(s))
fields.forEach(field => q.addOrderBy(getterByName(field)))
q
}
I am trying to write a help function to manipulate AST now. But that does not seem like the right solution.
Did not notice there is a version of orderBy that accepts a list of ExpressionNodes. Was able to solve it like this:
def samplesSorted(fields: List[String]) = {
from(Schema.samples)(s => select(s) orderBy(fields.map(buildOrderBy(s)))
}
def buildOrderBy(row: Row)(field: String): ExpressionNode = {
getterByName(row, field)
}
def getterByName(row: Row, field: String): String = field match {
case "Name" => row.name
case "Address" => row.address
}
Have not tried with fields of different types yet - implicits may not work in this case. But I could always call them explicitly.
Upd:
To do the same with descending order one could use a helper like this one:
def desc(node: ExpressionNode):ExpressionNode = new OrderByArg(node) {desc}
This works for me
def ord(dr: DataRow, name: String): ExpressionNode = if (orderAscending) {
dr.getterByName(name) asc
} else {
dr.getterByName(name) desc
}
case class DataRow(id: Long,
#Column("resource_id") resourceId: String,
def getterByName(name: String) = {
name match {
case "resource_id" => resourceId.~
case _ => id.~
}
}
}
from(DataSchema.dataRows) { dr =>
where(dr.id === id).select(dr).orderBy(ord(dr, filedName))
}.page(offset, limit)