Can't we use scala flatMap method on List of integers (i.e) List[Int]?
I am getting compile time error for the below code
object FlatMapExample {
def main(args:Array[String])
{
val numberList = List(1,2,3)
val mappedList = numberList.map { elem => elem*2 }
println(mappedList)
val flatMappedList = numberList.flatMap { elem => elem*2 }//compile time error
println(flatMappedList)
}
}
Compile time error:
type mismatch ; found: Int required :scala.collection.GenTraversableOnce[?]
flatMap() assumes you are returning a collection of values rather than a single element. Thus these would work:
val list = List(1,2,3)
list.flatMap(elem => List(elem * 2)) // List (2,4,6)
If you just want to multiply by two, use map.
Related
I need to ultimately build a schema from a CSV. I can read the CSV into data frame, and I've got a case class defined.
case class metadata_class (colname:String,datatype:String,length:Option[Int],precision:Option[int])
val foo = spark.read.format("csv").option("delimiter",",").option("header","true").schema(Encoders.product[metadata_class.schema).load("/path/to/file").as[metadata_file].toDF()
Now I'm trying to iterate through that data frame and build a list of StructFields. My current effort:
val sList: List[StructField] = List(
for (m <- foo.as[metadata_class].collect) {
StructField[m.colname,getType(m.datatype))
})
That gives me a type mismatch:
found : Unit
required: org.apache.spark.sql.types.StructField
for (m <- foo.as[metadata_class].collect) {
^
What am I doing wrong here? Or am I not even close?
There is not usual to use for-loop in scala. For loop has Unit return type, and in your code, result value of sList will be List[Unit]:
val sList: List[Unit] = List(
for (m <- foo.as[metadata_class].collect) {
StructField(m.colname, getType(m.datatype))
}
)
but you declared sList as List[StructField] this is the cause of compile error.
I suppose you should use map function instead of for loop for iterate on metadata_class objects and create StructFields from them:
val structFields: List[StructField] = foo.as[metadata_class]
.collect
.map(m => StructField(m.colname, getType(m.datatype)))
.toList
you will earn List[StructField] such way.
In scala language every statement is expression with return type, for-loop also and it return type is Unit.
read more about statements/expressions:
statement vs expression in scala
statements and expressions in scala
i have created a empty scala mutable list
import scala.collection.mutable.ListBuffer
val list_of_list : List[List[String]] = List.empty
i want to append elements to it as below
filtered_df.collect.map(
r => {
val val_list = List(r(0).toString,r(4).toString,r(5).toString)
list_of_list += val_list
}
)
error that i am getting is
Error:(113, 26) value += is not a member of List[List[String]]
Expression does not convert to assignment because receiver is not assignable.
list_of_list += val_list
Can someone help
Your declaration seems wrong:
val list_of_list : List[List[String]] = List.empty
means that you've declared scala.collection.immutable.List whose operations return a new list without changing the current.
To fix the error you need to change the outer List type to ListBuffer that you imported above the declaration as follows:
val list_of_list : ListBuffer[List[String]] = ListBuffer.empty
Also it looks like you don't to use map here unless you want to modify your data collected from DataFrame, so you can change it to foreach:
filtered_df.collect.foreach {
r => {
val val_list = List(r(0).toString,r(4).toString,r(5).toString)
list_of_list += val_list
}
}
Furthermore you can make it in a functional way without resorting to ListBuffer, by using immutable List and foldRight as follows:
val list_of_list: List[List[String]] =
filtered_df.collect.toList
.foldRight(List.empty[List[String]])((r, acc) => List(r(0).toString,r(4).toString,r(5).toString) :: acc)
toList is used to achieve a stack safety when calling foldRight, because it's not stack safe for Arrays
More info about foldLeft and foldRight
You have to change that val list_of_list to var list_of_list. That alone would not be enough as you also have to change the type of list_of_list into a mutable alternative.
How to conveniently convert Seq[Try[Option[String, Any]]] into Try[Option[Map[String, Any]]].
If any Try before convert throws an exception, the converted Try should throw as well.
Assuming that the input type has a tuple inside the Option then this should give you the result you want:
val in: Seq[Try[Option[(String, Any)]]] = ???
val out: Try[Option[Map[String,Any]]] = Try(Some(in.flatMap(_.get).toMap))
If any of the Trys is Failure then the outer Try will catch the exception raised by the get and return Failure
The Some is there to give the correct return type
The get extracts the Option from the Try (or raises an exception)
Using flatMap rather than map removes the Option wrapper, keeping all Some values and discaring None values, giving Seq[(String, Any)]
The toMap call converts the Seq to a Map
Here is something that's not very clean but may help get you started. It assumes Option[(String,Any)], returns the first Failure if there are any in the input Seq and just drops None elements.
foo.scala
package foo
import scala.util.{Try,Success,Failure}
object foo {
val x0 = Seq[Try[Option[(String, Any)]]]()
val x1 = Seq[Try[Option[(String, Any)]]](Success(Some(("A",1))), Success(None))
val x2 = Seq[Try[Option[(String, Any)]]](Success(Some(("A",1))), Success(Some(("B","two"))))
val x3 = Seq[Try[Option[(String, Any)]]](Success(Some(("A",1))), Success(Some(("B","two"))), Failure(new Exception("bad")))
def f(x: Seq[Try[Option[(String, Any)]]]) =
x.find( _.isFailure ).getOrElse( Success(Some(x.map( _.get ).filterNot( _.isEmpty ).map( _.get ).toMap)) )
}
Example session
bash-3.2$ scalac foo.scala
bash-3.2$ scala -classpath .
Welcome to Scala 2.13.1 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_66).
Type in expressions for evaluation. Or try :help.
scala> import foo.foo._
import foo.foo._
scala> f(x0)
res0: scala.util.Try[Option[Equals]] = Success(Some(Map()))
scala> f(x1)
res1: scala.util.Try[Option[Equals]] = Success(Some(Map(A -> 1)))
scala> f(x2)
res2: scala.util.Try[Option[Equals]] = Success(Some(Map(A -> 1, B -> two)))
scala> f(x3)
res3: scala.util.Try[Option[Equals]] = Failure(java.lang.Exception: bad)
scala> :quit
If you're willing to use a functional support library like Cats then there are two tricks that can help this along:
Many things like List and Try are traversable, which means that (if Cats's implicits are in scope) they have a sequence method that can swap two types, for example converting List[Try[T]] to Try[List[T]] (failing if any of the items in the list are failure).
Almost all of the container types support a map method that can operate on the contents of a container, so if you have a function from A to B then map can convert a Try[A] to a Try[B]. (In Cats language they are functors but the container-like types in the standard library generally have map already.)
Cats doesn't directly support Seq, so this answer is mostly in terms of List instead.
Given that type signature, you can iteratively sequence the item you have to in effect push the list type down one level in the type chain, then map over that container to work on its contents. That can look like:
import cats.implicits._
import scala.util._
def convert(listTryOptionPair: List[Try[Option[(String, Any)]]]): Try[
Option[Map[String, Any]]
] = {
val tryListOptionPair = listTryOptionPair.sequence
tryListOptionPair.map { listOptionPair =>
val optionListPair = listOptionPair.sequence
optionListPair.map { listPair =>
Map.from(listPair)
}
}
}
https://scastie.scala-lang.org/xbQ8ZbkoRSCXGDJX0PgJAQ has a slightly more complete example.
One way to approach this is by using a foldLeft:
// Let's say this is the object you're trying to convert
val seq: Seq[Try[Option[(String, Any)]]] = ???
seq.foldLeft(Try(Option(Map.empty[String, Any]))) {
case (acc, e) =>
for {
accOption <- acc
elemOption <- e
} yield elemOption match {
case Some(value) => accOption.map(_ + value)
case None => accOption
}
}
You start off with en empty Map. You then use a for comprehension to go through the current map and element and finally you add a new tuple in the map if present.
The following solutions is based on this answer to the point that almost makes the question a duplicate.
Method 1: Using recursion
def trySeqToMap1[X,Y](trySeq : Seq[Try[Option[(X, Y)]]]) : Try[Option[Map[X,Y]]] = {
def helper(it : Iterator[Try[Option[(X,Y)]]], m : Map[X,Y] = Map()) : Try[Option[Map[X,Y]]] = {
if(it.hasNext) {
val x = it.next()
if(x.isFailure)
Failure(x.failed.get)
else if(x.get.isDefined)
helper(it, m + (x.get.get._1-> x.get.get._2))
else
helper(it, m)
} else Success(Some(m))
}
helper(trySeq.iterator)
}
Method 2: directly pattern matching in case you are able to get a stream or a List instead:
def trySeqToMap2[X,Y](trySeq : LazyList[Try[Option[(X, Y)]]], m : Map[X,Y]= Map.empty[X,Y]) : Try[Option[Map[X,Y]]] =
trySeq match {
case Success(Some(h)) #:: tail => trySeqToMap2(tail, m + (h._1 -> h._2))
case Success(None) #:: tail => tail => trySeqToMap2(tail, m)
case Failure(f) #:: _ => Failure(f)
case _ => Success(Some(m))
}
note: this answer was previously using different method signatures. It has been updated to conform to the signature given in the question.
I have the following Scala code:
object Solution {
def getBestSolution(sumList: List[Int]): Int = {
return 0
}
def main(args: Array[String]) {
val t = readInt()
(0 until t).foreach({
val n = readInt()
val a = readLine().split(" ").map(_.toInt).toList
val sumList = a.scanLeft(0)(_ + _).tail.toList
//println(classOf[sumList])
println(sumList)
println(getBestSolution(sumList))
})
}
}
For it, I am getting this error:
file.scala:16: error: type mismatch;
found : Unit
required: Int => ?
println(getBestSolution(sumList))
^
one error found
Any idea what is causing this?
The argument you are passing to foreach is the result of executing the code block (which is a Unit), not a function.
Remove the outer parentheses (they do not really hurt anything, but are unnecessary and look ugly), and add _ => in the beginning:
(0 to t).foreach { _ =>
...
println(getBestSolution(sumList))
}
This is the proper syntax for creating an unnamed function. The stuff before => is the parameter list that the function accepts. In your case, you can just put an underscore there, because you do not need the value of the parameter. Or you could give it a name if you needed to do something with it, e.g.: (0 to t).foreach { x => println(x*x) }
you could have done it with simple for comprehension too instead of foreach
for(x <- 0 to t){
val n = readInt()
val a = readLine().split(" ").map(_.toInt).toList
val sumList = a.scanLeft(0)(_ + _).tail.toList
//println(classOf[sumList])
println(sumList)
println(getBestSolution(sumList))
}
To sum up, Programming in Scala book has pointed that Scala provides the for comprehension, which provides syntactically pleasing nesting of map, flatMap, and filter ... The for comprehension is not a looping construct, but is a syntactic construct the compiler reduces to map, flatMap, and filter.
I have a partition method that creates tuple of two sets of string.
def partition(i:Int) = {
dictionary.keySet.partition(dictionary(_)(i) == true)
}
I also have a map that maps integer to the return value from the partition method.
val m = Map[Int, (Set[String], Set[String])]()
for (i <- Range(0, getMaxIndex())) {
m(i) = partition(i)
}
The issue is that I have type mismatch error, but the error message does not make sense to me.
What might be wrong?
This is the code:
import scala.collection.mutable.Map
import scala.collection.{BitSet}
case class Partition(dictionary:Map[String, BitSet]) {
def max(x:Int, y:Int) = if (x > y) x else y
def partition(i:Int) = {
dictionary.keySet.partition(dictionary(_)(i) == true)
}
def getMaxIndex() = {
val values = dictionary.values
(0 /: values) ((m, bs) => max(m, bs.last))
}
def get() = {
val m = Map[Int, (Set[String], Set[String])]()
for (i <- Range(0, getMaxIndex())) {
m(i) = partition(i)
}
m
}
}
When I compile your example, the error is clear:
<console>:64: error: type mismatch;
found : (scala.collection.Set[String], scala.collection.Set[String])
required: (scala.collection.immutable.Set[String], scala.collection.immutable.Set[String])
m(i) = partition(i)
^
Looking into the API, the keySet method of a mutable map does not guarantee that the returned set is immutable. Compare this with keySet on an immutable Map—it does indeed return an immutable set.
Therefore, you could either
use an immutable Map and a var
force the result of your partition method to return an immutable set (e.g. toSet)
define the value type of your map to be collection.Set instead of Predef.Set which is an alias for collection.immtuable.Set.
To clarify these types, it helps to specify an explicit return type for your public methods (get and partition)