I am having a requirement where I have to loop over a list and do create Map[String,String]. Here the header has values as list like below:
val headersMap = scala.collection.mutable.Map[String, String]()
try {
val payloadHeaders = collectorPayload.headers
for (values <- payloadHeaders.toList) {
for (value <- values) {
val header = value.split(":").map(_.trim)
headersMap += (header(0) -> header(1))
headersMap += ("Content-Type" -> "application/json; charset=UTF-8")
}
}
} catch {
case e: Exception => {
logger.error("Collector Payload extraction error with : " + e.getMessage)
}
}
Is there any better way to handle this any map or flatMap way?
Don't use mutable collections or variables (just pretend they don't exist util you run into a use case where you positively cannot do without them ... it won't be soon).
Also generally avoid using loops (because they kinda assume and promote mutability and side effects), you'll need them even less often than mutable collections.
collectorPayload
.headers
.iterator
.flatMap(_.split(":").map(_.trim))
.map { case Array(a,b) => a -> b }
.toMap + ("Content-Type" -> "application/json; charset=UTF-8")
Related
I need to use many Maps in my project so I wonder which way is more efficient:
val map = mutable.Map[Int, Int] = mutable.Map.empty
for (_ <- 0 until big_number)
{
// do something with map
map.clear()
}
or
for (_ <- 0 until big_number)
{
val map = mutable.Map[Int, Int] = mutable.Map.empty
// do something with map
}
to use in terms of time and memory?
Well, my formal answer would always be depends. As you need to benchmark your own scenario, and see what fits better for your scenario. I'll provide an example how you can try benchmarking your own code. Let's start with writing a measuring method:
def measure(name: String, f: () => Unit): Unit = {
val l = System.currentTimeMillis()
println(name + ": " + (System.currentTimeMillis() - l))
f()
println(name + ": " + (System.currentTimeMillis() - l))
}
Let's assume that in each iteration we need to insert into the map one key-value pair, and then to print it:
Await.result(Future.sequence(Seq(Future {
measure("inner", () => {
for (i <- 0 until 10) {
val map2 = mutable.Map.empty[Int, Int]
map2(i) = i
println(map2)
}
})
},
Future {
measure("outer", () => {
val map1 = mutable.Map.empty[Int, Int]
for (i <- 0 until 10) {
map1(i) = i
println(map1)
map1.clear()
}
})
})), 10.seconds)
The output in this case, is almost always equal between the inner and the outer. Please note that in this case I run the two options in parallel, as if I wouldn't the first one always takes significantly more time, no matter which one of then is first.
Therefore, we can conclude, that in this case they are almost the same.
But, if for example I add an immutable option:
Future {
measure("immutable", () => {
for (i <- 0 until 10) {
val map1 = Map[Int, Int](i -> i)
println(map1)
}
})
}
it always ends up first. This makes sense because immutable collections are much more performant than the mutables.
For better performance tests you probably need to use some third parties, such as scalameter, or others that exists.
I have an Object
object Constants {
val getA = "example.a.test"
val getB = "example.b.test"
val getC = "example.c.test"
.
.
.
}
I have another class where I'm accessing these values after importing the class in an if-else loop
if(str == "A") {
println(Constants.getA)
}
else if (str == "B") {
println(Constants.getB)
} else {
println(Constants.getC)
}
// and so on...
NOTE: This is just a sample code I have but the if-else loops get complicated. Is there a way to simplify this by passing the "str" variable directly to the object like "Constants.get$str" or something simpler? I get Cyclomatic complexity Scala check style warning
You can use pattern matching and create a new function in Constant.
def getString(str: String) = {
str match {
case "A" => "example.a.test"
case "B" => "example.b.test"
case "C" => "example.c.test"
case _ => "Wrong input"
}
}
you can use a key/value object for handle your code
for example use Map :
object Constants {
val get = Map("A" -> "example.a.test", "B" -> "example.b.test", "C" -> "example.c.test" , ...)
}
and you can use it by
println( get("A"))
instead of all if else loop which you had .
you can even iterate on the keys or values too :
get.keys.foreach{ i =>
print( "Key = " + i )
println(" Value = " + get(i) )}
i think this way could be simpler and i hope the answer is useful.
I'm following the tutorial from Alvin Alexander to use Loan Pattern
Here is the code what I use -
val year = 2016
val nationalData = {
val source = io.Source.fromFile(s"resources/Babynames/names/yob$year.txt")
// names is iterator of String, split() gives the array
//.toArray & toSeq is a slow process compare to .toSet // .toSeq gives Stream Closed error
val names = source.getLines().filter(_.nonEmpty).map(_.split(",")(0)).toSet
source.close()
names
// println(names.mkString(","))
}
println("Names " + nationalData)
val info = for (stateFile <- new java.io.File("resources/Babynames/namesbystate").list(); if stateFile.endsWith(".TXT")) yield {
val source = io.Source.fromFile("resources/Babynames/namesbystate/" + stateFile)
val names = source.getLines().filter(_.nonEmpty).map(_.split(",")).
filter(a => a(2).toInt == year).map(a => a(3)).toArray // .toSet
source.close()
(stateFile.take(2), names)
}
println(info(0)._2.size + " names from state "+ info(0)._1)
println(info(1)._2.size + " names from state "+ info(1)._1)
for ((state, sname) <- info) {
println("State: " +state + " Coverage of name in "+ year+" "+ sname.count(n => nationalData.contains(n)).toDouble / nationalData.size) // Set doesn't have length method
}
This is how I applied readTextFile, readTextFileWithTry on the above code to learn/experiment Loan Pattern in the above code
def using[A <: { def close(): Unit }, B](resource: A)(f: A => B): B =
try {
f(resource)
} finally {
resource.close()
}
def readTextFile(filename: String): Option[List[String]] = {
try {
val lines = using(fromFile(filename)) { source =>
(for (line <- source.getLines) yield line).toList
}
Some(lines)
} catch {
case e: Exception => None
}
}
def readTextFileWithTry(filename: String): Try[List[String]] = {
Try {
val lines = using(fromFile(filename)) { source =>
(for (line <- source.getLines) yield line).toList
}
lines
}
}
val year = 2016
val data = readTextFile(s"resources/Babynames/names/yob$year.txt") match {
case Some(lines) =>
val n = lines.filter(_.nonEmpty).map(_.split(",")(0)).toSet
println(n)
case None => println("couldn't read file")
}
val data1 = readTextFileWithTry("resources/Babynames/namesbystate")
data1 match {
case Success(lines) => {
val info = for (stateFile <- data1; if stateFile.endsWith(".TXT")) yield {
val source = fromFile("resources/Babynames/namesbystate/" + stateFile)
val names = source.getLines().filter(_.nonEmpty).map(_.split(",")).
filter(a => a(2).toInt == year).map(a => a(3)).toArray // .toSet
(stateFile.take(2), names)
println(names)
}
}
But in the second case, readTextFileWithTry, I am getting the following error -
Failed, message is: java.io.FileNotFoundException: resources\Babynames\namesbystate (Access is denied)
I guess the reason for the failure is from SO what I understand -
I am trying to open the same file on each iteration of the for loop
Apart from that, I have few concerns regarding how I use -
Is it the good way to use? Can some help me how can I use the TRY on multiple occasions?
I tried to change the return type of readTextFileWithTry like Option[A] or Set/Map or Scala Collection to apply higher-order functions later on that. but not able to succeed. Not sure that is a good practice or not.
How can I use higher-order functions in Success case, as there are multiple operations and in Success case the code blocks get bigger? I can't use any field outside of Success case.
Can someone help me to understand?
I think that you problem has nothing to do with "I am trying to open the same file on each iteration of the for loop" and it is actually the same as in the accepted answer
Unfortunately you didn't provide stack trace so it is not clear on which line this happens. I would guess that the falling call is
val data1 = readTextFileWithTry("resources/Babynames/namesbystate")
And looking at your first code sample:
val info = for (stateFile <- new java.io.File("resources/Babynames/namesbystate").list(); if stateFile.endsWith(".TXT")) yield {
it looks like the path "resources/Babynames/namesbystate" points to a directory. But in your second example you are trying to read it as a file and this is the reason for the error. It comes from the fact that your readTextFileWithTry is not a valid substitute for java.io.File.list call. And File.list doesn't need a wrapper because it doesn't use any intermediate closeable/disposable entity.
P.S. it might make more sense to use File.list(FilenameFilter filter) instead of if stateFile.endsWith(".TXT"))
Im new to scala and searched a lot for the solution.
I'm querying the database and storing the value of the http request parsed as a json4s object in response. I wait for the response and parse the json.
val refService = url("http://url//)
val response = Http(refService OK dispatch.as.json4s.Json)
var checkVal :Boolean = true
val json = Await.result(response, 30 seconds)
val data = json \ "data"
I want to run a loop and check if the value of "name" is present in the data returned. If present I want to break and assign checkVal to false. So far I have this:
for {
JObject(obj) <- data
JField("nameValue", JString(t)) <- obj //nameValue is the column name in the returned data
} yield {checkVal= if (t == name){ break }
else
true
}
Eclipse is giving me the following error: type mismatch; found : List[Unit] required:
List[String]
Please advice. Thank you.
One of your problems is that you have different return types in yield: if t==name, return type is the type of break, and if t!=name return type is Boolean.
In scala you don't have break operator, this behaviour is achieved using breakable construct and calling break() method which actually throws an exception to exit from breakable block. Also you can use if statements in for body to filter you results:
import scala.util.control.Breaks._
breakable {
for {
JObject(obj) <- data
JField("nameValue", JString(t)) <- obj
if t == name
} yield {
checkVal = false
break()
}
}
UPDATE:
I used this imperative approach because you are new to scala, but it's not scala way. IMHO you should stick to #Imm code in comments to your question.
I actually don't like using pattern matching in for loops as if for some reason data is not a JObject it won't be handled well. I prefer an approach like below.
data match {
case JObject(fields) => fields.exists{
case (name:String,value:JString) => name == "nameValue" && value.s == "name"
case _ => false
}
case _ => false // handle error as not a JObject
}
Edit: revised to include your matches.
I would suggest to use exists as it is lazy on all collection members.
code:-
val list= Map(
"nameValue1" -> 1,
"nameValue2" -> 2,
"nameValue3" -> 3,
"nameValue4" -> 4,
"nameValue5" -> 5
)
val requiredHeader = "nameValue2"
var keyvalue:Int=0
list.exists(p=>{ if(p._1.equalsIgnoreCase(requiredHeader))keyvalue=p._2;p._1.equalsIgnoreCase(requiredHeader) })
if(keyvalue!=0){
//header present
}else{
//header doesn't exit
}
I'm parallelising over a collection to count the number same item values in a List. The list in this case is uniqueSetOfLinks :
for (iListVal <- uniqueSetOfLinks.par) {
try {
val num : Int = listOfLinks.count(_.equalsIgnoreCase(iListVal))
linkTotals + iListVal -> num
}
catch {
case e : Exception => {
e.printStackTrace()
}
}
}
linkTotals is an immutable Map. To gain a reference to the total number of links do I need to update linkTotals so that it is immutable ?
I can then do something like :
linkTotals.put(iListVal, num)
You can't update immutable collection, all you can do is to combine immutable collection with addition element to get new immutable collection, like this:
val newLinkTotals = linkTotals + (iListVal -> num)
In case of collection you could create new collection of pairs and than add all pairs to the map:
val optPairs =
for (iListVal <- uniqueSetOfLinks.par)
yield
try {
val num : Int = listOfLinks.count(_.equalsIgnoreCase(iListVal))
Some(iListVal -> num)
}
catch {
case e : Exception => e.printStackTrace()
None
}
val newLinkTotals = linkTotals ++ optPairs.flatten // for non-empty initial map
val map = optPairs.flatten.toMap // in case there is no initial map
Note that you are using parallel collections (.par), so you should not use mutable state, like linkTotals += iListVal -> num.
Possible variation of #senia's answer (got rid of explicit flatten):
val optPairs =
(for {
iListVal <- uniqueSetOfLinks.par
count <- {
try
Some(listOfLinks.count(_.equalsIgnoreCase(iListVal)))
catch {
case e: Exception =>
e.printStackTrace()
None
}
}
} yield iListVal -> count) toMap
I think that you need some form of MapReduce in order to have parallel number of items estimation.
In your problem you already have all unique links. The partial intermediate result of map is simply a pair. And "reduce" is just toMap. So you can simply par-map the link to pair (link-> count) and then finally construct a map:
def count(iListVal:String) = listOfLinks.count(_.equalsIgnoreCase(iListVal))
val listOfPairs = uniqueSetOfLinks.par.map(iListVal => Try( (iListVal, count(iListVal)) ))
("map" operation is par-map)
Then remove exceptions:
val clearListOfPairs = listOfPairs.flatMap(_.toOption)
And then simply convert it to a map ("reduce"):
val linkTotals = clearListOfPairs.toMap
(if you need to check for exceptions, use Try.failure)