I am working on a PlayFramework application written in Scala.
Problem is that in a rest controller I need a list of elements (books) and for each element list of its subelements (chapters).
Book repository:
def findAll(): Future[Seq[Book]]
Chapter repository:
def findByBookId(bookId: UUID): Future[Seq[Chapter]]
I wanted to do something like
val books = bookRepository.findAll
val result = for {
bookList <- books
book <- bookList
chapters <- chapterRepository.findByBookdId(book.id)
} yield (book, chapters)
I want to have a tuple of book and its chapters so I can latter map it to a json. What I am doing wrong, because I get error:
[error] required: scala.collection.GenTraversableOnce[?]
Or what would be a better approach how to iterate over future of collection and for each element load another future of collection?
See if this gets you close to what you're after.
// found books (future)
val fBooks :Future[Seq[Book]] = bookRepository.findAll()
// chapters in fBooks order (future)
val fChptrs :Future[Seq[Seq[Chapter]]] = fBooks.flatMap {books =>
Future.sequence(books.map(book =>findByBookId(book.id)))
}
// combine book with its chapters
val result :Future[Seq[(Book, Seq[Chapter])]] = for {
books <- fBooks
chptrs <- fChptrs
} yield books zip chptrs
You can only use one type of monad in a for comprehension, since it is just syntactic sugar for flatMap and map.
See here the whole description: https://stackoverflow.com/a/33402543/2750966
You should use the same type throughout the for-comprehension.
You can't have books:Future[Seq[Books] and bookList: Seq[Book] in same for-comprehension.
Also you can convert Seq[Future[Something]] to Future[Seq[Something]] using Future.sequence
So, something like this should work
val res: Future[(Seq[Book], Seq[Chapter])] =
for {
bookList <- books
chapters <- Future.sequence(bookList.map(book => findByBookId(book.id)))
} yield (bookList, chapters.flatten)
Related
I have the following scenario:
a List[Building] collection I'll call buildings
a def getAddress(building: Building): Future[Address] function
the result should be a List[BuildingWithAddress] collection
I want to iterate over the List[Building] and for each element call getAddress and wait for it to complete so that I can create a new object of type BuildingWithAddress and store it in a collection that I'll then return to the caller.
I thought I'd use a for-comprehension but it turns out that something along the line of this won't really work:
for {
building <- listOfBuildings
address <- getAddress(building)
buildingWithAddress = BuildingWithAddress(name = building.name, town = address.town)
} yield buildingWithAddress
I also thought on using flatMap to iterate over the list and then do the same for the address but the types are different and it won't work.
I tried with a forEach but then again the forEach isn't waiting on the Future to complete.
What's the solution for such a simple use case?
You cannot usually combine different monads in a single for comprehension (except for scala's collection-likes). Here you want to combine the Future and List monads, which cannot be done this way.
If you want to do this in a "sequential" way (waiting for previous future operations to finish before starting new ones) you need to use a ListT monad transformer from scalaz (or cats) like this:
import scalaz.ListT
import scalaz.std.scalaFuture._
implicit executor: ExecutionContext = ...
def fromList[A](x: List[A]) = ListT(Future.successful(x))
def single[A](x: Future[A]) = ListT(x.map(List(_)))
(for {
building <- fromList(listOfBuildings)
address <- single(getAddress(building))
buildingWithAddress = BuildingWithAddress(name = building.name, town = address.town)
} yield buildingWithAddress).run
Which will result in a Future[List[...]] as you need.
Alternative solution if you are fine with calling the getAddress functions in parallel:
Future.traverse(listOfBuildings)(building =>
getAddress(building).map(address =>
BuildingWithAddress(name = building.name, town = address.town)))
This is traverses the List "applicatively" (meaning: in parallel).
use Future.sequence
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
val l = List(1,2,3)
val futureList: Seq[Future[Int]] = l.map(e=>Future(e))
//conversion List[Future[T]] => Future[List[T]]
val singleFuture: Future[Seq[Int]] = Future.sequence(futureList)
singleFuture.map(_.length)
Probably this's simple problem, but I begin my adventure with spark.
Problem: I'd like to get following structure (Expected result) in spark. Now I have following structure.
title1, {word11, word12, word13 ...}
title2, {word12, word22, word23 ...}
Data are stored in Dataset[(String, Seq[String])]
Excepted result
I would like to get Tuple [word, title]
word11, {title1} word12, {title1}
What I do
1. Make (title, seq[word1,word2,word,3])
docs.mapPartitions { iter =>
iter.map {
case (title, contents) => {
val textToLemmas: Seq[String] = toText(....)
(title, textToLemmas)
}
}
}
I tried use .map to transform my structure to Tuple, but can't do it.
I tried to iterate through all the elements, but then I can not return type
Thanks for answer.
This should work:
val result = dataSet.flatMap { case (title, words) => words.map((_, title)) }
Another solution is to call the explode function like this :
import org.apache.spark.sql.functions.explode
dataset.withColumn("_2", explode("_2")).as[(String, String)]
Hope this help you, Best Regrads.
I'm surprised no one offered a solution with Scala's for-comprehension (that gets "desugared" to flatMap and map as in Yuval Itzchakov's answer at compile time).
When you see a series of flatMap and map (possibly with filter) that's Scala's for-comprehension.
So the following:
val result = dataSet.flatMap { case (title, words) => words.map((_, title)) }
is equivalent to the following:
val result = for {
(title, words) <- dataSet
w <- words
} yield (w, title)
After all, that's why we enjoy flexibility of Scala, isn't it?
I am new to Scala from Java so the functional programming thing is still a bit difficult for me to understand. I have a project in Play framework. I need to query the database to get rows with ids and display them in a html template.
Here is my code
def search(query: String) = Action.async{ request =>
val result = SearchEngine.searchResult(query)
val docs = result.map(DocumentService.getDocumentByID(_).map(doc => doc))
val futures = Future.sequence(docs)
futures.map{documents =>
Ok(views.html.results(documents.flatten))
}
}
getDocumentByID returns a Future[Options[Document]] object, but my results template takes Array[Document] so I have tried to no avail to transform the Future[Options[Document]] to Array[Document]
The current code I have is the closest I have been, but it still does not compile. This is the error:
Error:(36, -1) Play 2 Compiler:
found : Array[scala.concurrent.Future[Option[models.Document]]]
required: M[scala.concurrent.Future[A]]
Try to collect only the Somes from the Future returned by the getDocumentByID
val docs = result.map { res =>
val f: Future[Option[Document]] = DocumentService.getDocumentByID(res)
f.collect { case Some(doc) => doc }
}.toList
val futures = Future.seqence(docs) //notice that docs is converted to list from array in the previous line
General suggestion
Do not use Arrays. Arrays are mutable and they do not grow dynamically.
So it is advisable to avoid using Array in concurrent/parallel code.
Let's say I have some Repository API where I have wrapped the transactions in a (Scalaz) Reader monad. Now I want to run computations over the results, and save the results back into the Repository. I tried something like:
type UOW[A] = Reader[Transaction, A]
object Record1Repo {
override def findAll: UOW[Seq[Record1]] = Reader(t => {
...
})
}
...
repo.run {
for {
all: Seq[Record1] <- Record1Repo.findAll
record: Record <- all
encoding: Encoding <- Processor.encode(record)
_ <- Record2Repo.save(Record2(encoding))
} yield {
logger.info(s"processed record id=${record.id}")
}}
But it falls apart with the futile attempt to map over the results in record <- all.
I'm quite new to this type of functional programming and couldn't find how express my intention properly. Any suggestions is welcome.
It fails because you are breaking out of the Reader monad.
You start with a Reader and then you extract from a Seq so this cannot be translated in a flatMap/map chain within the Reader structure.
Basically the same question has been asked about a year ago for slick 2.x (scala slick one-to-many collections). I'm wondering if there has any progression been made with the release of reactive slick.
Let's say for example we have three tables. library, book and library_to_book where a library has many books. What I want is a list of libraries with their books. In scala this would be something like Seq[(Library, Seq[Book])]. The query I have is as follows:
val q = (for {
l <- libraries
ltb <- libraryToBooks if l.id === ltb.libraryId
b <- books if ltb.bookId === b.id
} yield (l, b)
db.run(q.result).map( result => ??? )
results in this case is of type Seq[(Library, Book)]. How do I have to change my query to get a result of type Seq[(Library, Seq[Book])] instead? What is the "slick way" of writing such queries?
IMO your code looks fine. It really depends on what feels more readable to you. Alternatively, you can use join as well:
val findBooksQuery = libraries
.join(libraryToBooks).on(_.id === _.libraryId)
.join(books).on(_.id === _._2.bookId)
.result
val action = (for {
booksResult <- findBooksQuery
} yield {
booksResult.map { row =>
val (libraryTableRow, libraryToBooksTableRow) = row._1
val booksTableRow = row._2
// TODO: Access all data from the rows and construct desired DS
}
}
db.run(action)
You can then do a groupBy on a particular key to get the kind of data structure you are looking for. In this case, it would be more evolved as it is join across three tables. Example, add following to your query:
val findBooksQuery = libraries
.join(libraryToBooks).on(_.id === _.libraryId)
.join(books).on(_.id === _._2.bookId)
// To group by libraries.id
.groupBy(_._1.id)
.result
To what you want to map to, db.run returns a Future(of something), a Future[Seq[(Library, Seq[Book])]] in your case. When mapping over a future you have access to the Seq and you can transform it to something else to get a new Future.