I try to run the example Scala code of CoGroup function, which is provided in the Flink website , but it throw error "value map is not a member of Object".
Here is my code
val iVals: DataSet[(String, Int)] = env.fromCollection(Seq(("a",1),("b",2),("c",3)))
val dVals: DataSet[(String, Int)] = env.fromCollection(Seq(("a",11),("b",22)))
val output = iVals.coGroup(dVals).where(0).equalTo(0) {
(iVals, dVals, out: Collector[Double]) =>
val ints = iVals map { _._2 } toSet
for (dVal <- dVals) {
for (i <- ints) {
out.collect(dVal._2 * i)
}
}
}
output.print()
I don't know what cause the error or is there any library I miss to import? Thanks.
Have you tried adding the type annotations for iVals and dVals? It seems that Scala is inferring the type Object, hence the error. (Why, I don't know).
What I mean is:
(iVals: Iterator[(String, Int)], dVals: Iterator[(String, Int)], out: Collector[Double]) =>
Related
How to conveniently convert Seq[Try[Option[String, Any]]] into Try[Option[Map[String, Any]]].
If any Try before convert throws an exception, the converted Try should throw as well.
Assuming that the input type has a tuple inside the Option then this should give you the result you want:
val in: Seq[Try[Option[(String, Any)]]] = ???
val out: Try[Option[Map[String,Any]]] = Try(Some(in.flatMap(_.get).toMap))
If any of the Trys is Failure then the outer Try will catch the exception raised by the get and return Failure
The Some is there to give the correct return type
The get extracts the Option from the Try (or raises an exception)
Using flatMap rather than map removes the Option wrapper, keeping all Some values and discaring None values, giving Seq[(String, Any)]
The toMap call converts the Seq to a Map
Here is something that's not very clean but may help get you started. It assumes Option[(String,Any)], returns the first Failure if there are any in the input Seq and just drops None elements.
foo.scala
package foo
import scala.util.{Try,Success,Failure}
object foo {
val x0 = Seq[Try[Option[(String, Any)]]]()
val x1 = Seq[Try[Option[(String, Any)]]](Success(Some(("A",1))), Success(None))
val x2 = Seq[Try[Option[(String, Any)]]](Success(Some(("A",1))), Success(Some(("B","two"))))
val x3 = Seq[Try[Option[(String, Any)]]](Success(Some(("A",1))), Success(Some(("B","two"))), Failure(new Exception("bad")))
def f(x: Seq[Try[Option[(String, Any)]]]) =
x.find( _.isFailure ).getOrElse( Success(Some(x.map( _.get ).filterNot( _.isEmpty ).map( _.get ).toMap)) )
}
Example session
bash-3.2$ scalac foo.scala
bash-3.2$ scala -classpath .
Welcome to Scala 2.13.1 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_66).
Type in expressions for evaluation. Or try :help.
scala> import foo.foo._
import foo.foo._
scala> f(x0)
res0: scala.util.Try[Option[Equals]] = Success(Some(Map()))
scala> f(x1)
res1: scala.util.Try[Option[Equals]] = Success(Some(Map(A -> 1)))
scala> f(x2)
res2: scala.util.Try[Option[Equals]] = Success(Some(Map(A -> 1, B -> two)))
scala> f(x3)
res3: scala.util.Try[Option[Equals]] = Failure(java.lang.Exception: bad)
scala> :quit
If you're willing to use a functional support library like Cats then there are two tricks that can help this along:
Many things like List and Try are traversable, which means that (if Cats's implicits are in scope) they have a sequence method that can swap two types, for example converting List[Try[T]] to Try[List[T]] (failing if any of the items in the list are failure).
Almost all of the container types support a map method that can operate on the contents of a container, so if you have a function from A to B then map can convert a Try[A] to a Try[B]. (In Cats language they are functors but the container-like types in the standard library generally have map already.)
Cats doesn't directly support Seq, so this answer is mostly in terms of List instead.
Given that type signature, you can iteratively sequence the item you have to in effect push the list type down one level in the type chain, then map over that container to work on its contents. That can look like:
import cats.implicits._
import scala.util._
def convert(listTryOptionPair: List[Try[Option[(String, Any)]]]): Try[
Option[Map[String, Any]]
] = {
val tryListOptionPair = listTryOptionPair.sequence
tryListOptionPair.map { listOptionPair =>
val optionListPair = listOptionPair.sequence
optionListPair.map { listPair =>
Map.from(listPair)
}
}
}
https://scastie.scala-lang.org/xbQ8ZbkoRSCXGDJX0PgJAQ has a slightly more complete example.
One way to approach this is by using a foldLeft:
// Let's say this is the object you're trying to convert
val seq: Seq[Try[Option[(String, Any)]]] = ???
seq.foldLeft(Try(Option(Map.empty[String, Any]))) {
case (acc, e) =>
for {
accOption <- acc
elemOption <- e
} yield elemOption match {
case Some(value) => accOption.map(_ + value)
case None => accOption
}
}
You start off with en empty Map. You then use a for comprehension to go through the current map and element and finally you add a new tuple in the map if present.
The following solutions is based on this answer to the point that almost makes the question a duplicate.
Method 1: Using recursion
def trySeqToMap1[X,Y](trySeq : Seq[Try[Option[(X, Y)]]]) : Try[Option[Map[X,Y]]] = {
def helper(it : Iterator[Try[Option[(X,Y)]]], m : Map[X,Y] = Map()) : Try[Option[Map[X,Y]]] = {
if(it.hasNext) {
val x = it.next()
if(x.isFailure)
Failure(x.failed.get)
else if(x.get.isDefined)
helper(it, m + (x.get.get._1-> x.get.get._2))
else
helper(it, m)
} else Success(Some(m))
}
helper(trySeq.iterator)
}
Method 2: directly pattern matching in case you are able to get a stream or a List instead:
def trySeqToMap2[X,Y](trySeq : LazyList[Try[Option[(X, Y)]]], m : Map[X,Y]= Map.empty[X,Y]) : Try[Option[Map[X,Y]]] =
trySeq match {
case Success(Some(h)) #:: tail => trySeqToMap2(tail, m + (h._1 -> h._2))
case Success(None) #:: tail => tail => trySeqToMap2(tail, m)
case Failure(f) #:: _ => Failure(f)
case _ => Success(Some(m))
}
note: this answer was previously using different method signatures. It has been updated to conform to the signature given in the question.
I am new to scala - spark and loaded my dataset in RDD . here is my sample data set
scala> flightdata.collect
res39: Array[(String, Int)] = Array((DFW,11956), (DTW,588), (SEA,607), (JFK,1595), (SJC,327), (ORD,4664), (PHX,4993), (STL,661),
from the above dataset , i need to find total sum . Hence i written like this
scala> flightdata.values.sum
res40: Double = 445827.0
scala> flightdata.map(_._2).reduce( (a,b) => a + b)
res41: Int = 445827
Both value.sum and map using reduce is giving the right answer. But i am trying to rewrite the same code tuple with reduce.
scala> flightdata.reduce( (s1,s2) => s1._2 + s2._2)
<console>:26: error: type mismatch;
found : Int
required: (String, Int)
flightdata.reduce( (s1,s2) => s1._2 + s2._2)
it is causing error. type mismatch. why it is causing type mismatch error
It happens because you try to combine two tuples, but have integer as result.
You should return tuple ("", s1._2 + s2._2) instead of s1._2 + s2._2.
Given the following scala code:
val mapAll= Map("A"-> 1.toShort, "B"-> 2.toShort)
val map1 = Map("A"-> 1.toShort)
val map2 = mapAll.map(x => (x._2, map1.getOrElse(x._1, -1.toShort)))
In this case，the type of map2 shown in IDEA is Map[Short, Int].
What I need is Map[Short, Short]
However, if put the -1 into "()" like this"
val map2 = mapAll.map(x => (x._2, map1.getOrElse(x._1, (-1).toShort)))
It's just fine. the type of map2 shown in IDEA is Map[Short, Short]
So， What's the difference between "(-1).toShort" and "-1.toShort"
There's no difference. You're probably tricked by IntelliJ's bad type inference in this case. You can see this when compiling scalac with the -Xprint:typer flag:
val map2: scala.collection.immutable.Map[Short,Short] =
mapAll.map[(Short, Short), scala.collection.immutable.Map[Short,Short]](
((x: (String, Short)) => scala.Tuple2.apply[Short, Short](
x._2, map1.getOrElse[Short](x._1, -1.toShort))))
(immutable.this.Map.canBuildFrom[Short, Short]);
We can also verify this by simplifying the example:
val shortOne = -1.toShort
val shortTwo = (-1).toShort
And compiling with -Xprint:jvm to see the final emitted step:
val shortOne: Short = -1.toShort();
val shortTwo: Short = -1.toShort();
You can see both are identical.
On Short, - is defined like this : def unary_- : Int
Happens, -1.toShort is understood as -(1.toShort) by IDEA, of type Int, and (-1).toShort is correctly typed Short, since this is the type of toShort which is now "called" after the - instead of before.
I want to return both a Future[Seq[String]] from a method and the length of that Seq[String] as well. Currently I'm building the Future[Seq[String]] using a mapping function from another Future[T].
Is there any way to do this without awaiting for the Future?
You can map over the current Future to create a new one with the new data added to the type.
val fss: Future[Seq[String]] = Future(Seq("a","b","c"))
val x: Future[(Seq[String],Int)] = fss.map(ss => (ss, ss.length))
If you somehow know what the length of the Seq will be without actually waiting for it, then something like this;
val t: Future[T] = ???
def foo: (Int, Future[Seq[String]]) = {
val length = 42 // ???
val fut: Future[Seq[String]] = t map { v =>
genSeqOfLength42(v)
}
(length, fut)
}
If you don't, then you will have to return Future[(Int, Seq[String])] as jwvh said, or you can easily get the length later in the calling function.
Can't we use scala flatMap method on List of integers (i.e) List[Int]?
I am getting compile time error for the below code
object FlatMapExample {
def main(args:Array[String])
{
val numberList = List(1,2,3)
val mappedList = numberList.map { elem => elem*2 }
println(mappedList)
val flatMappedList = numberList.flatMap { elem => elem*2 }//compile time error
println(flatMappedList)
}
}
Compile time error:
type mismatch ; found: Int required :scala.collection.GenTraversableOnce[?]
flatMap() assumes you are returning a collection of values rather than a single element. Thus these would work:
val list = List(1,2,3)
list.flatMap(elem => List(elem * 2)) // List (2,4,6)
If you just want to multiply by two, use map.