I use this code to read file into memory :
val lines = Source.fromFile(fileToRead, "utf-8").getLines
To iterate over some of the lines I use :
lines.take(linesToReadFromDataFile).foreach(line => {
Sometimes I may want to iterate all lines :
lines.foreach(line => {
To determines if to real all of the lines I could use a boolean 'useAlllines' and do something like :
if(useAllLines)
lines.foreach(line => {
else
lines.take(linesToReadFromDataFile).foreach(line => {
Using Scala is there a better way of achieving this ?
I guess this will be enough:
val toIterate =
if(useAllLines)
lines
else
lines.take(linesToReadFromDataFile)
for ( line <- toIterate ) {
...
}
You could also combine useAllLines and linesToReadFromDataFile in a single variable of type Option[Int]:
val toIterate = optionLinesToReadFromDataFile.map{ lines.take(_) }.getOrElse(lines)
lines.take(if (useAllLines) lines.length else linesToReadFromDataFile).foreach(
Related
How do I simplify this loop to some function like foreach or map or other thing with Scala? I want to put hitsArray inside that filter shipList.filter.
val hitsArray: Array[String] = T.split(" ");
for (hit <- hitsArray) {
shipSize = shipList.length
shipList = shipList.filter(!_.equalsIgnoreCase(hit))
}
if (shipList.length == 0) {
shipSunk = shipSunk + 1
} else if (shipList.length < shipSize) {
shipHit = shipHit + 1
}
To be fair, I don't understand why you are calling shipSize = shipList.length as you don't use it anywhere.
T.split(" ").foreach{ hit =>
shipList = shipList.filter(!_.equalsIgnoreCase(hit))
}
which gets you to where you want to go. I've made it 3 lines because you want to emphasize you're working via side effect in that foreach. That said, I don't see any advantage to making it a one-liner. What you had before was perfectly readable.
Something like this maybe?
shipList.filter(ship => T.split(" ").forall(!_.equalsIgnoreCase(ship)))
Although cleaner if shipList is already all lower case:
shipList.filterNot(T.split(" ").map(_.toLowerCase) contains _)
Or if your T is large, move it outside the loop:
val hits = T.split(" ").map(_.toLowerCase)
shipList.filterNot(hits contains _)
I am subscript to a message feed for a number of fields, I need to set the values from the feed to the domain object and have code like below:
if (map.contains(quoteBidPriceAcronym)) {
quote.bid.price = Some(map.get(quoteBidPriceAcronym).get.asInstanceOf[Number].doubleValue());
quote.changed = true;
}
if (map.contains(quoteBidSizeAcronym)) {
quote.bid.size = Some(sizeMultipler() * map.get(quoteBidSizeAcronym).get.asInstanceOf[Number].intValue());
quote.changed = true;
}
if (map.contains(quoteBidNumAcronym)) {
quote.bid.num = Some(map.get(quoteBidNumAcronym).get.asInstanceOf[Number].shortValue());
quote.changed = true;
}
if (map.contains(quoteAskPriceAcronym)) {
quote.ask.price = Some(map.get(quoteAskPriceAcronym).get.asInstanceOf[Number].doubleValue());
quote.changed = true;
}
if (map.contains(quoteAskSizeAcronym)) {
quote.ask.size = Some(sizeMultipler() * map.get(quoteAskSizeAcronym).get.asInstanceOf[Number].intValue());
quote.changed = true;
}
if (map.contains(quoteAskNumAcronym)) {
quote.ask.num = Some(map.get(quoteAskNumAcronym).get.asInstanceOf[Number].shortValue());
quote.changed = true;
}
if (map.contains(quoteExchTimeAcronym)) {
quote.exchtime = getExchTime(String.valueOf(map.get(quoteExchTimeAcronym).get));
}
It look pretty redundant, any suggestion to improve it?
You can do something like:
map.get(quoteBidPriceAcronym).map { item =>
quote.bid.price = item.map(_.asInstanceOf[Number].doubleValue())
quote.changed = true
}
Other issues might be better to fix outside. E.g. why map[quoteBidPriceAcronym] is storing an Option, if your code assumes it's not going to be None?
Something like this perhaps?
val handlers = Map[String, Number => Unit] (
quoteBidPriceAcronym -> { n => quote.bid.price = Some(n.doubleValue) },
quoteBidSizeAcronym -> { n => quote.bid.size = Some(sizeMultipler() * n.intValue },
etc. ...
)
for {
(k,handler) <- handlers
values <- map.get(k).toSeq
quote.chanded = true
_ = handler(n.asInstanceof[Number])
}
Personally, I don't like code changing an object state (quote) but this is a question on Scala, not functional programming.
That said I would reverse the way you are using you map map keys. Instead of checking whether a value exists to perform some action, I'd have a map from your keys to actions and I'd iterate over your map elements.
e.g (assuming map is of the type Map[String, Any]):
val actions: Map[String, PartialFunction[Any, Unit]] = Map(
(quoteBidPriceAcronym, {case n: Number => quote.bid.price = Some(n.doubleValue())}),
(quoteBidSizeAcronym, {case n: Number => quote.bid.size = Some(sizeMultipler() * n.doubleValue())}),
...
...
)
for((k,v) <- map; action <- actions.get(k); _ <- action.lift(v))
quote.changed = true;
The for construct here iterates over map key-values, then (next level of iteration, over the possible action available for the key. If an action is found, which is a partial function, it gets lifted to make it a function from Any to Option[Unit]. That way, you can iterate in an additional inner level so quote.changed = true is only run when the action is defined for v.
I have the following Scala snippet from my code. I am not able to convert it into functional style. I could do it at other places in my code but not able to change the below one to functional. Issue is once the code exhausts all pattern matching options, then only it should send back "NA". Following code is doing that, but it's not in functional style (for-yield)
var matches = new ListBuffer[List[String]]()
for (line <- caselist){
var count = 0
for (pat <- pattern if (!pat.findAllIn(line).isEmpty)){
count += 1
matches += pat.findAllIn(line).toList
}
if (count == 0){
matches += List("NA")
}
}
return matches.toList
}
Your question is not entirely complete, so I can't be sure, but I believe the following will do the job:
for {
line <- caselist
matches = pattern.map(_.findAllIn(line).toList)
} yield matches.flatten match {
case Nil => List("NA")
case ms => ms
}
This should do the job. Using foreach and filter to generate the matches and checking to make sure there is a match for each line will work.
caseList.foreach{ line =>
val results = pattern.foreach ( pat => pat.findAllIn(line).toList )
val filteredResults = results.filter( ! _.isEmpty )
if ( filteredResults.isEmpty ) List("NA")
else filteredResults
}
Functional doesn't mean you can't have intermediate named values.
Given a Future[Seq[Widget]], where Widget contains a amount : Int property, I'd like to return a Seq[Widget] but for only those Widgets whose amount value is greater than 100. I believe the for { if … } yield { } construct will give me what I want but am unsure how to filter through the Sequence. I have:
val myWidgetFuture : Future[Seq[Widget]] = ...
for {
widgetSeq <- myWidgetFuture
if (??? amount > 100) <— what to put here?
} yield {
widgetSeq
}
If there's a clean non-yield way of doing this that will also work for me.
You don't even need yield. Use map.
val myWidgetFuture: Future[Seq[Widget]] = ???
myWidgetFuture map { ws => ws filter (_.amount > 100) }
If you want to use for … yield with an if filter, you'll need to use two fors:
for {
widgetSeq <- myWidgetFuture
} yield for {
widget <- widgetSeq
if widget.amount > 100
} yield widget
I read file from HDFS, which contains x1,x2,y1,y2 representing a envelope in JTS.
I would like to use those data to build STRtree in foreach.
val inputData = sc.textFile(inputDataPath).cache()
val strtree = new STRtree
inputData.foreach(line => {val array = line.split(",").map(_.toDouble);val e = new Envelope(array(0),array(1),array(2),array(3)) ;
println("envelope is " + e);
strtree.insert(e,
new Rectangle(array(0),array(1),array(2),array(3)))})
As you can see, I also print the e object.
To my surprise, when I log the size of strtree, it is zero! It seems that insert method make no senses here.
By the way, if I write hard code some test data line by line, the strtree can be built well.
One more thing, those project is packed into jar and submitted in the spark-shell.
So, why does the method in foreach not work ?
You will have to collect() to do this:
inputData.collect().foreach(line => {
... // your code
})
You can do this (for avoiding collecting all data):
val pairs = inputData.map(line => {
val array = line.split(",").map(_.toDouble);
val e = new Envelope(array(0),array(1),array(2),array(3)) ;
println("envelope is " + e);
(e, new Rectangle(array(0),array(1),array(2),array(3)))
}
pairs.collect().foreach(pair => {
strtree.insert(pair._1, pair._2)
}
Use .map() instead of .foreach() and reassign the outcome.
Foreach does not return the outcome of applyied function. It can be used for sending data somewhere, storing to db, printing, and so on.