I am trying to get this query right but getting errors. First, searchUser returns a sequence of matching UserEntries that contains unique id for user and for each of the userId, there is a second query obtains some other user info + address from another table.
Code:
def searchUsers(pattern: String) = auth.SecuredAction.async {
implicit request =>
usersService.searchUser(pattern) flatMap { usrList =>
for {
u <- usrList
ui <- usersService.getUsersWithAddress(u.id)
} yield {
Ok(views.html.UserList(ui))
}
}
}
Signatures for the APIs used:
def searchUser(pattern: String): Future[Seq[UserEntry]] = ...
def getUsersWithAddress(userId: Long): Future[Seq[(UserProfile, Seq[String])]] = ...
Error:
[error] modules/www/app/controllers/Dashboard.scala:68: type mismatch;
[error] found : scala.concurrent.Future[play.api.mvc.Result]
[error] required: scala.collection.GenTraversableOnce[?]
[error] ui <- usersService.getUsersWithAddress(u.id)
[error] ^
[error] one error found
[error] (www/compile:compileIncremental) Compilation failed
If I comment out line "u <- usrList" and hardcode a userid for the next line like "ui <- usersService.getUsersWithAddress(1L)" it works. Any idea what I am missing?
When you use multiple generators in a for comprehension, the monads have to be of the same type. E.g. you can't:
scala> for{ x <- Some("hi"); y <- List(1,2,3) } yield (x,y)
<console>:11: error: type mismatch;
found : List[(String, Int)]
required: Option[?]
for{ x <- Some("hi"); y <- List(1,2,3) } yield (x,y)
^
What you can do is convert one or the other to match the correct type. For the above example, that would be:
scala> for{ x <- Some("hi").toSeq; y <- List(1,2,3) } yield (x,y)
res2: Seq[(String, Int)] = List((hi,1), (hi,2), (hi,3))
In your particular case, one of your generators is a GenTraversableOnce, and the other is a Future. You can probably use Future.successful(theList) to get two futures. See the answer here for example:
Unable to use for comprehension to map over List within Future
Based on #Brian's answer arrived at a solution.. following worked (could not enter formatter block of code as comment - so adding as answer):
usersService.searchUser(pattern) flatMap { usrList =>
val q = for {
u <- usrList
} yield (usersService.getUserWithAddress(u.id))
val r = Future.sequence(q)
r map { ps =>
Ok(views.html.UserList(ps))
}
}
For comprehension accumulated the Futures and then the sequence was flattened and then mapped. Hope this is how it is done!
Note: also I had to change the signature of getUserWithAddress to X instead of Seq[X]
Related
I am not very versed in Scala and sincerily I found the documentation hard to try to figure this out, but I was wondering if someone could explain to me why the difference in compilation of the following statements:
I can easily iterate over a set of strings and yield the elements.
scala> for(name <- Set("asd", "123")) yield name
val res2: scala.collection.immutable.Set[String] = Set(asd, 123)
But i can't do it inline if the Set is inside an Option
scala> for(names <- Some(Set("asd", "123")); name <- names) yield (name)
^
error: type mismatch;
found : scala.collection.immutable.Set[String]
required: Option[?]
It happens because of for-yield is just syntactic sugar for flatMap, map and withFilter functions. So, your code:
for(names <- Some(Set("asd", "123")); name <- names) yield (name)
actually is the same as:
Some(Set("asd", "123")).flatMap{ names: Set[String] =>
names.flatMap{ name: String =>
name // return type is String, but Set[(of some )B] is expected
}
}// return type is Set[String] but Option[(of some) C] is expected
look at the Option flatMap function:
#inline final def flatMap[B](f: A => Option[B]): Option[B]
but your f returns Set[String]
as a result, compiler tell you about type missmatch (Set[String] != Option[?]):
error: type mismatch;
found : scala.collection.immutable.Set[String]
required: Option[?]
you should remember about type of the first statement in for-yield construction. In your case it's names <- Some(Set("asd", "123")). It has type Option[Set[String]], so you should use only Option[T] in x <- yourNextStatement lines (x should has Option[T] type).
In conclusion:
Be careful with mixing different container types in for-yield constructions. If you have some problems, just try to unwrap your for-yield into combination of flatMap, map, withFilter functions.
If you want to mix containers in for-yeild, you should start another for-yield for each-container type. For example:
for {
names <- Some(Set("asd", "123"))
// will return Option[String]
reducedNames <- (for {
name <- names // here will be used Seq flatMap function, not Option
} yield (name + "some_suffix"))
.reduceLeftOption(_ + _) // here we will get Option[String] from Seq[String]
} yield reducedNames // will return Option[String]
What do you want to happen for None? Assuming "nothing", you could do
val optionalSet = Some(Set("asd", "123"))
for (names <- optionalSet.getOrElse(Set.empty); name <- names) yield name
object Main {
val list = List[Long]()
val map1 = Map[Long, List[Long]]()
val map2 = Map[Long, Long]()
def main(args: Array[String]): Unit = {
for {
a: Long <- list
b: List[Long] <- map1.get(a)
c: Long <- b
d: Long <- map2.get(c)
} yield d
}
}
The type of a,b,c,d has been declared just for easy to analysis the code.
Then I get the unexpected compilation error:
Error:(10, 15) type mismatch;
found : List[Long]
required: Option[?]
c: Long <- b
The type of b is a List, I think c should mean each item in the List b
Why the compiler expect b should be a Option something?
The problem is that you are mixing options with lists. If you convert your option to a List it works:
b: List[Long] <- map1.get(a).toList
Remember that a for comprehension is just syntactic sugar for flatMap. This is actually performing a flatMapon the result of map1.get(a):
b: List[Long] <- map1.get(a)
c: Long <- b
which expects an option as a result. To avoid confusion you should always use the same type you started with, which is List in this case.
Another option would be to skip b entirely and directly flatten the result of the first map:
a: Long <- list
c: Long <- map1.get(a).toList.flatten
d: Long <- map2.get(c)
I'm trying to generate an RDD from a List and another RDD using scala and spark. The idea is to take a list of values, and generate an index containing all the entries of the original dataset that contains each value.
Here's the code that I'm trying
def mcveInvertIndex(foos: List[String], bars: RDD[Int]): RDD[(String, Iterable[Int])] = {
// filter function
def hasVal(foo: String)(bar: Int): Boolean =
foo.toInt == bar
// call to sc.parallelize to ensure that an RDD is returned
sc parallelize(
foos map (_ match {
case (s: String) => (s, bars filter hasVal(s))
})
)
}
Unfortunately this does not compile in sbt
> compile
[info] Compiling 1 Scala source to $TARGETDIR/target/scala-2.11/classes...
[error] $TARGETDIR/src/main/scala/wikipedia/WikipediaRanking.scala:56: type mismatch;
[error] found : List[(String, org.apache.spark.rdd.RDD[Int])]
[error] required: Seq[(String, Iterable[Int])]
[error] Error occurred in an application involving default arguments.
[error] foos map (_ match {
[error] ^
[error] one error found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 1 s, completed Mar 11, 2017 7:11:31 PM
I really don't understand the errors that I'm getting. List is a subclass of Seq, and I presume that RDDs are a subclass of Iterable. Is there something obvious that I've missed?
Here is my solution with for-comprehension (should use less memory than cartesian product)
def mcveInvertIndex(foos: List[String],
bars: RDD[Int]): RDD[(String, Iterable[Int])] =
{
// filter function
def hasVal(foo: String, bar: Int): Boolean =
foo.toInt == bar
// Producing RDD[(String, Iterable[Int])]
(for {
bar <- bars // it's important to have RDD
// at first position of for-comprehesion
// to produce the correct result type
foo <- foos
if hasVal(foo, bar)
} yield (foo, bar)).groupByKey()
}
As mentioned in the comment, RDD is not an Iterable, so you have to combine the two in some way and then aggregate them. This is my quick solution, although there might be a more efficient way:
def mcveInvertIndex(foos: List[String], bars: RDD[Int]): RDD[(String, Iterable[Int])] = {
sc.makeRDD(foos)
.cartesian(bars)
.keyBy(x=>x._1)
.aggregateByKey(Iterable.empty[Int])(
(agg: Iterable[Int], currVal: (String, Int))=>{
if(currVal._1.toInt != currVal._2) agg
else currVal._2 +: agg.toList
},
_ ++ _
)
}
I am trying to do several dependent Slick/DB calls and then display the resulting data within a twirl template.
def show(slug: String) = Action.async { implicit rs =>
for {
f <- fooDAO.findBySlug(slug) // f is of type Option[foo]
fid <- f.flatMap(a => a.id.map(b => b)) // fid is of type Long
b <- barDAO.findByFooId(fid) // b is of type Seq[bar]
} yield {
f.map {
case Some(f) => Ok(views.html.foobar(f, b))
case _ => NotFound
}
}
}
I first need to get the "ID" to then be able to query other relevant data. The compiler is now producing this error:
play.sbt.PlayExceptions$CompilationException: Compilation error[type mismatch;
found : scala.concurrent.Future[Option[play.api.mvc.Result]]
required: Option[?]]
Any help would be greatly appreciated.
There is a fundamental flaw in your code, in that you're mixing in the same comprehension an Option and a Seq
A for-comprehension is expected to work on the same "container" type, which will be the resulting representation of the yield
e.g. if you combine several Options you get an Option, if you combine Seqs you get a Seq.
In this case you can overcome the problem by converting the Option (foo) to a Seq (which will be empty if the foo is None and have 1 element if not).
The end result would be
val results: Seq[(Foo, Bar)] =
for {
f <- fooDAO.findBySlug(slug).toSeq // f is of type Seq[Foo]
b <- barDAO.findByFooId(f.id) // b is of type Seq[Bar]
} yield (f, b)
But I guess this is not what you need. I suppose you want to get all Bars associated with the retrieved Foo, if any, and present it with your template. If no Foo is present for the slug, you want a NotFound.
We can do it like this
def show(slug: String) = Action.async { implicit rs =>
val f = fooDAO.findBySlug(slug) // f is of type Option[Foo]
f.fold(
NotFound,
foo => Ok(views.html.foobar(foo, barDAO.findByFooId(foo.id))
)
}
You can make it more explicit by defining a supporting method
def show(slug: String) = Action.async { implicit rs =>
def barsOf(f: Foo): Seq[Bar] = barDAO.findByFooId(f.id)
val f = fooDAO.findBySlug(slug) // f is of type Option[Foo]
f.fold(
NotFound,
foo => Ok(views.html.foobar(foo, barsOf(foo))
)
}
It's a bit tricky understanding what you're trying to achieve here, but if the whole thing is predicated on the findbySlug returning a Future[Option[Foo]] and the eventual outcome being a NotFound if that Option is a None, then your yield should probably just be:
...
} yield {
f.fold(NotFound)(foo => Ok(views.html.foobar(foo, b)))
}
Option[T] is a fantastic type for data-retrieval and control-flow, but pattern-matching on it is almost never the right approach. The use of fold feels nicely succinct for the task.
Given a List of JsObject's, I'd like to convert it to a Map[String, JsValue], and then iterate through each map's key-value pair, making a JsObject per pair. Finally, I'd like to put all of these JsObject's into a List[JsObject].
Below is my attempt with a for expression.
def getObjMap(obj: JsObject): Option[Map[String, JsValue]] = obj match {
case JsObject(fields) => Some(fields.toMap)
case _ => None
}
def createFields(x: (String, JsValue)): JsObject = Json.obj("n" -> x._1,
"v" -> x._2)
val objects: List[JsObject] = foo()
val res: List[JsObject] = for {
obj <- objects
map <- getObjMap(obj)
mapField <- map
} yield(createFields(mapField))
However, I'm getting a compile-time error on the mapField <- map line.
[error] ...\myApp\app\services\Test.scala:65: type mismatch;
[error] found : scala.collection.immutable.Iterable[play.api.libs.
json.JsObject]
[error] required: Option[?]
I think the problem is coming from the fact that the for comprehension starts by working over a List, then ends up working over an Option (from getObjMap). The type of construct needs to remain constant across the whole for comprehension. Calling .toList on the result of the getObjMap call fixes this:
val res: List[JsObject] = for {
obj <- objects
map <- getObjMap(obj).toList
mapField <- map
} yield(createFields(mapField))