i have created a empty scala mutable list
import scala.collection.mutable.ListBuffer
val list_of_list : List[List[String]] = List.empty
i want to append elements to it as below
filtered_df.collect.map(
r => {
val val_list = List(r(0).toString,r(4).toString,r(5).toString)
list_of_list += val_list
}
)
error that i am getting is
Error:(113, 26) value += is not a member of List[List[String]]
Expression does not convert to assignment because receiver is not assignable.
list_of_list += val_list
Can someone help
Your declaration seems wrong:
val list_of_list : List[List[String]] = List.empty
means that you've declared scala.collection.immutable.List whose operations return a new list without changing the current.
To fix the error you need to change the outer List type to ListBuffer that you imported above the declaration as follows:
val list_of_list : ListBuffer[List[String]] = ListBuffer.empty
Also it looks like you don't to use map here unless you want to modify your data collected from DataFrame, so you can change it to foreach:
filtered_df.collect.foreach {
r => {
val val_list = List(r(0).toString,r(4).toString,r(5).toString)
list_of_list += val_list
}
}
Furthermore you can make it in a functional way without resorting to ListBuffer, by using immutable List and foldRight as follows:
val list_of_list: List[List[String]] =
filtered_df.collect.toList
.foldRight(List.empty[List[String]])((r, acc) => List(r(0).toString,r(4).toString,r(5).toString) :: acc)
toList is used to achieve a stack safety when calling foldRight, because it's not stack safe for Arrays
More info about foldLeft and foldRight
You have to change that val list_of_list to var list_of_list. That alone would not be enough as you also have to change the type of list_of_list into a mutable alternative.
Related
I need to ultimately build a schema from a CSV. I can read the CSV into data frame, and I've got a case class defined.
case class metadata_class (colname:String,datatype:String,length:Option[Int],precision:Option[int])
val foo = spark.read.format("csv").option("delimiter",",").option("header","true").schema(Encoders.product[metadata_class.schema).load("/path/to/file").as[metadata_file].toDF()
Now I'm trying to iterate through that data frame and build a list of StructFields. My current effort:
val sList: List[StructField] = List(
for (m <- foo.as[metadata_class].collect) {
StructField[m.colname,getType(m.datatype))
})
That gives me a type mismatch:
found : Unit
required: org.apache.spark.sql.types.StructField
for (m <- foo.as[metadata_class].collect) {
^
What am I doing wrong here? Or am I not even close?
There is not usual to use for-loop in scala. For loop has Unit return type, and in your code, result value of sList will be List[Unit]:
val sList: List[Unit] = List(
for (m <- foo.as[metadata_class].collect) {
StructField(m.colname, getType(m.datatype))
}
)
but you declared sList as List[StructField] this is the cause of compile error.
I suppose you should use map function instead of for loop for iterate on metadata_class objects and create StructFields from them:
val structFields: List[StructField] = foo.as[metadata_class]
.collect
.map(m => StructField(m.colname, getType(m.datatype)))
.toList
you will earn List[StructField] such way.
In scala language every statement is expression with return type, for-loop also and it return type is Unit.
read more about statements/expressions:
statement vs expression in scala
statements and expressions in scala
I am trying to use a Scala HashMap like below, and when I try to operate on the HashMap, I get type mismatches. Besides using Java HashMap, how can I write this?
import scala.collection.mutable._
object Solution {
def twoSum(nums: Array[Int], target: Int): Array[Int] = {
var lookupTable = new HashMap()
for(i <- nums.indices) {
if (lookupTable.contains(target - nums(i)))
return Array(lookupTable(i), i)
lookupTable.put(nums(i), i)
}
throw new Exception
}
}
In the absence of a type ascription, an empty HashMap is a HashMap[Nothing, Nothing], which, because mutable HashMaps are invariant means you have a collection which you can't put anything into or get anything out of.
There are thus two ways to fix this:
You can use a mutable HashMap[Int, Int]:
val lookupTable = HashMap[Int, Int]()
Alternatively, you can use an immutable HashMap, which is covariant in the value type and technically invariant in the key type, but there's a way around that:
var lookupTable = scala.collection.immutable.HashMap()
for (i <- nums.indices) {
if (lookupTable.contains(target - nums(i))) return Array(lookupTable(i), i)
else lookupTable = lookupTable ++ Seq((nums(i), i))
}
val mymap= collection.mutable.Map.empty[String,Seq[String]]
mymap("key") = collection.mutable.ListBuffer("a","b")
mymap.get("key") += "c"
The last line to append to the list buffer is giving error. How the append can be done ?
When you run the code in the scala console:
→$scala
scala> val mymap= collection.mutable.Map.empty[String,Seq[String]]
mymap: scala.collection.mutable.Map[String,Seq[String]] = Map()
scala> mymap("key") = collection.mutable.ListBuffer("a","b")
scala> mymap.get("key")
res1: Option[Seq[String]] = Some(ListBuffer(a, b))
You'll see that mymap.get("key") is an optional type. You can't add a string to the optional type.
Additionally, since you typed mymap to Seq[String], Seq[String] does not have a += operator taking in a String.
The following works:
val mymap= collection.mutable.Map.empty[String,collection.mutable.ListBuffer[String]]
mymap("key") = collection.mutable.ListBuffer("a","b")
mymap.get("key").map(_ += "c")
Using the .map function will take advantage of the optional type and prevent noSuchElementException as Łukasz noted.
To deal with your problems one at a time:
Map.get returns an Option[T] and Option does not provide a += or + method.
Even if you use Map.apply (mymap("key")) the return type of apply will be V (in this case Seq) regardless of what the actual concrete type is (Vector, List, Set, etc.). Seq does not provide a += method, and its + method expects another Seq.
Given that, to get what you want you need to declare the type of the Map to be a mutable type:
import collection.mutable.ListBuffer
val mymap= collection.mutable.Map.empty[String,ListBuffer[String]]
mymap("key") = ListBuffer("a","b")
mymap("key") += "c"
will work as you expect it to.
If you really want to have immutable value, then something like this should also work:
val mymap= collection.mutable.Map.empty[String,Seq[String]]
mymap("key") = Vector("a","b")
val oldValue = mymap.get("key").getOrElse(Vector[String]())
mymap("key") = oldValue :+ "c"
I used Vector here, because adding elements to the end of List is unefficient by design.
val indices: List[Int] = List()
val featValues: List[Double] = List()
for (f <- feat) {
val q = f.split(':')
if (q.length == 2) {
println(q.mkString("\n")) // works fine, displays info
indices :+ (q(0).toInt)
featValues :+ (q(1).toDouble)
}
}
println(indices.mkString("\n") + indices.length) // prints nothing and 0?
indices and featValues are not being filled. I'm at a loss here.
You cannot append anything to an immutable data structure such as List stored in a val (immutable named slot).
What your code is doing is creating a new list every time with one element appended, and then throwing it away (by not doing anything with it) — the :+ method on lists does not modify the list in place (even when it's a mutable list such as ArrayBuffer) but always returns a new list.
In order to achieve what you want, the quickest way (as opposed to the right way) is either to use a var (typically preferred):
var xs = List.empty[Int]
xs :+= 123 // same as `xs = xs :+ 123`
or a val containing a mutable collection:
import scala.collection.mutable.ArrayBuffer
val buf = ArrayBuffer.empty[Int]
buf += 123
However, if you really want to make your code idiomatic, you should instead just use a functional approach:
val indiciesAndFeatVals = feat.map { f =>
val Array(q0, q1) = f.split(':') // pattern matching in action
(q0.toInt, q1.toDouble)
}
which will give you a sequence of pairs, which you can then unzip to 2 separate collections:
val (indicies, featVals) = indiciesAndFeatVals.unzip
This approach will avoid the use of any mutable data structures as well as vars (i.e. mutable slots).
Suppose I have
import scala.collection.immutable.TreeMap
val tree = new TreeMap[String, List[String]]
Now after above declaration, I want to assign key "k1" to List("foo", "bar")
and then how do i get or read back the key "k1" and also read back non-existent key "k2"?
what happens if I try to read non-existent key "k2" ?
The best way to "mutate" the immutable map is by referring to it in a variable (var as opposed to val):
var tree = TreeMap.empty[String, List[String]]
tree += ("k1" -> List("foo", "bar")) //a += b is sugar for "c = a + b; a = c"
It can be accessed directly using the apply method, where scala syntactic sugar kicks in so you can just access using parens:
val l = tree("k1") //equivalent to tree.apply("k1")
However, I rarely access maps like this because the method will throw a MatchError is the key is not present. Use get instead, which returns an Option[V] where V is the value-type:
val l = tree.get("k1") //returns Option[List[String]] = Some(List("foo", "bar"))
val m = tree.get("k2") //returns Option[List[String]] = None
In this case, the value returned for an absent key is None. What can I do with an optional result? Well, you can make use of methods map, flatMap, filter, collect and getOrElse. Try and avoid pattern-matching on it, or using the Option.get method directly!
For example:
val wordLen : List[Int] = tree.get("k1").map(l => l.map(_.length)) getOrElse Nil
EDIT: one way of building a Map without declaring it as a var, and assuming you are doing this by transforming some separate collection, is to do it via a fold. For example:
//coll is some collection class CC[A]
//f : A => (K, V)
val m = (TreeMap.empty[K, V] /: coll) { (tree, c) => tree + f(c) }
This may not be possible for your use case