I am working in Scala with java libraries. One of these libraries returns a list of lists. I want to flatten the list.
Example:
import scala.collection.JavaConverters._
var parentList : util.List[util.List[Int]] = null
parentList = new util.ArrayList[util.List[Int]]
parentList.asScala.flatten // error
I have used asScala converter but I'm still meeting an error.
You need to call .asScala on every inner list :
scala> parentList.asScala.map(_.asScala)
res0: scala.collection.mutable.Buffer[scala.collection.mutable.Buffer[Int]] = ArrayBuffer()
scala> parentList.asScala.map(_.asScala).flatten
res1: scala.collection.mutable.Buffer[Int] = ArrayBuffer()
Note that calling .map and then .flatten can be done in one step using .flatMap :
scala> parentList.asScala.flatMap(_.asScala)
res2: scala.collection.mutable.Buffer[Int] = ArrayBuffer()
You also need to convert the inner List[Int]:
parentList.asScala.flatMap(_.asScala)
Try like this
import scala.jdk.CollectionConverters._
parentList.asScala.flatMap.map(_.toSeq)
This will do the trick.
Related
I think I understand the rules of implicit returns but I can't figure out why splithead is not being set. This code is run via
val m = new TaxiModel(sc, file)
and then I expect
m.splithead
to give me an array strings. Note head is an array of strings.
import org.apache.spark.SparkContext
import org.apache.spark.rdd.RDD
class TaxiModel(sc: SparkContext, dat: String) {
val rawData = sc.textFile(dat)
val head = rawData.take(10)
val splithead = head.slice(1,11).foreach(splitData)
def splitData(dat: String): Array[String] = {
val splits = dat.split("\",\"")
val split0 = splits(0).substring(1, splits(0).length)
val split8 = splits(8).substring(0, splits(8).length - 1)
Array(split0).union(splits.slice(1, 8)).union(Array(split8))
}
}
foreach just evaluates expression, and do not collect any data while iterating. You probably need map or flatMap (see docs here)
head.slice(1,11).map(splitData) // gives you Array[Array[String]]
head.slice(1,11).flatMap(splitData) // gives you Array[String]
Consider also a for comprehension (which desugars in this case into flatMap),
for (s <- head.slice(1,11)) yield splitData(s)
Note also that Scala strings are equipped with ordered collections methods, thus
splits(0).substring(1, splits(0).length)
proves equivalent to any of the following
splits(0).drop(1)
splits(0).tail
In scala, how can I initialize a scala collection from a Java iterable, in a clean idiomatic way?
Here's somewhat lame code taking a less functional approach for that:
var collection = Seq[MyClass]()
while (iterator.hasNext) {
val asArray: Array[String] = iterator.next.toArray
val val2 = asArray(2)
val val3 = asArray(3)
collection = collection :+ new MyClass(val2, val3)
}
How can initialization of a collection from a Java iterable take place more idiomatically?
import scala.collection.JavaConverters._
val collection = iterator.asScala.map{ x =>
val asArray = x.toArray
new MyClass(asArray(2), asArray(3))
}.toIndexedSeq
Scala can convert to and from Java collections seamlessly, provided you have imported the conversion helpers like below:
import scala.collection.JavaConversions._
val jl = new java.util.ArrayList[String]()
jl.add("Hello")
jl.add("There")
val collection = j1.map{ x => new MyClass(x(2), x(3)) }.toList
I have an conf/application.conf setting like
mongodb.replicaSetSeeds = ["bobk-mbp.local:27017","bobk-mbp.local:27018"]
I'm pulling it out in my code like (the actual extraction is a little different, but this is the gist of it)
val replicaSetSeeds = Play.current.configuration.getStringList("mongodb.replicaSetSeeds")
val listOfString: List[String] = replicaSetSeeds.getOrElse(List("localhost"))
but the compiler hates me
type mismatch; found : Object required: List[String]
The signature of getStringList is
def getStringList(path: String): Option[java.util.List[String]]
How do I handle the None case here or is my problem List[String] is not the same as List[java.util.String]?
Give this a shot:
import collection.JavaConversions._
val optList:Option[List[String]] = Play.current.configuration.getStringList("mongodb.replicaSetSeeds").map(_.toList)
val list = optList.getOrElse(List("localhost"))
There are multiple things going on here. First, you need to import the JavaConversions implicits because what's being returned is an Option[java.util.List[String]] and we want that to be a scala List instead. By doing the map(_.toList), I'm forcing the implicit conversion to kick in and get me an Option[List[String]] and from there things are pretty straight forward.
In play 2.5, you need to use dependency injection, the following works well for me:
1) in your class inject Configuration
class Application #Inject()(
configuration: play.api.Configuration
) ...
2) in your method
import scala.collection.JavaConversions._
val optlist = configuration.getStringList("mongodb.replicaSetSeeds").map{_.toList}
val list = optList.getOrElse(List("localhost"))
How does scala.collection.JavaConversions supercede the answers given in Stack Overflow question Iterating over Java collections in Scala (it doesn't work because the "jcl" package is gone) and in Iterating over Map with Scala (it doesn't work for me in a complicated test which I'll try to boil down and post here later).
The latter is actually a Scala Map question, but I think I need to know both answers in order to iterate over a java.util.Map.
In 2.8, you import scala.collection.JavaConversions._ and use as a Scala map. Here's an example (in 2.8.0.RC1):
scala> val jmap:java.util.Map[String,String] = new java.util.HashMap[String,String]
jmap: java.util.Map[String,String] = {}
scala> jmap.put("Hi","there")
res0: String = null
scala> jmap.put("So","long")
res1: String = null
scala> jmap.put("Never","mind")
res2: String = null
scala> import scala.collection.JavaConversions._
import scala.collection.JavaConversions._
scala> jmap.foreach(kv => println(kv._1 + " -> " + kv._2))
Hi -> there
Never -> mind
So -> long
scala> jmap.keys.map(_.toUpperCase).foreach(println)
HI
NEVER
SO
If you specifically want a Scala iterator, use jmap.iterator (after the conversions import).
I was trying to pull environment variables into a scala script using java Iterators and / or Enumerations and realised that Dr Frankenstein might claim parentage, so I hacked the following from the ugly tree instead:
import java.util.Map.Entry
import System._
val propSet = getProperties().entrySet().toArray()
val props = (0 until propSet.size).foldLeft(Map[String, String]()){(m, i) =>
val e = propSet(i).asInstanceOf[Entry[String, String]]
m + (e.getKey() -> e.getValue())
}
For example to print the said same environment
props.keySet.toList.sortWith(_ < _).foreach{k =>
println(k+(" " * (30 - k.length))+" = "+props(k))
}
Please, please don't set about polishing this t$#d, just show me the scala gem that I'm convinced exists for this situation (i.e java Properties --> scala.Map), thanks in advance ;#)
Scala 2.10.3
import scala.collection.JavaConverters._
//Create a variable to store the properties in
val props = new Properties
//Open a file stream to read the file
val fileStream = new FileInputStream(new File(fileName))
props.load(fileStream)
fileStream.close()
//Print the contents of the properties file as a map
println(props.asScala.toMap)
Scala 2.7:
val props = Map() ++ scala.collection.jcl.Conversions.convertMap(System.getProperties).elements
Though that needs some typecasting. Let me work on it a bit more.
val props = Map() ++ scala.collection.jcl.Conversions.convertMap(System.getProperties).elements.asInstanceOf[Iterator[(String, String)]]
Ok, that was easy. Let me work on 2.8 now...
import scala.collection.JavaConversions.asMap
val props = System.getProperties() : scala.collection.mutable.Map[AnyRef, AnyRef] // or
val props = System.getProperties().asInstanceOf[java.util.Map[String, String]] : scala.collection.mutable.Map[String, String] // way too many repetitions of types
val props = asMap(System.getProperties().asInstanceOf[java.util.Map[String, String]])
The verbosity, of course, can be decreased with a couple of imports. First of all, note that Map will be a mutable map on 2.8. On the bright side, if you convert back the map, you'll get the original object.
Now, I have no clue why Properties implements Map<Object, Object>, given that the javadocs clearly state that key and value are String, but there you go. Having to typecast this makes the implicit option much less attractive. This being the case, the alternative is the most concise of them.
EDIT
Scala 2.8 just acquired an implicit conversion from Properties to mutable.Map[String,String], which makes most of that code moot.
In Scala 2.9.1 this is solved by implicit conversions inside collection.JavaConversions._ . The other answers use deprecated functions. The details are documented here. This is a relevant snippet out of that page:
scala> import collection.JavaConversions._
import collection.JavaConversions._
scala> import collection.mutable._
import collection.mutable._
scala> val jul: java.util.List[Int] = ArrayBuffer(1, 2, 3)
jul: java.util.List[Int] = [1, 2, 3]
scala> val buf: Seq[Int] = jul
buf: scala.collection.mutable.Seq[Int] = ArrayBuffer(1, 2, 3)
scala> val m: java.util.Map[String, Int] = HashMap("abc" -> 1, "hello" -> 2)
m: java.util.Map[String,Int] = {hello=2, abc=1}
Getting from a mutable map to an immutable map is a matter of calling toMap on it.
In Scala 2.8.1 you can do it with asScalaMap(m : java.util.Map[A, B]) in a more concise way:
var props = asScalaMap(System.getProperties())
props.keySet.toList.sortWith(_ < _).foreach { k =>
println(k + (" " * (30 - k.length)) + " = " + props(k))
}
In Scala 2.13.2:
import scala.jdk.javaapi.CollectionConverters._
val props = asScala(System.getProperties)
Looks like in the most recent version of Scala (2.10.2 as of the time of this answer), the preferred way to do this is using the explicit .asScala from scala.collection.JavaConverters:
import scala.collection.JavaConverters._
val props = System.getProperties().asScala
assert(props.isInstanceOf[Map[String, String]])