Related
In Scala, what would be the right way of selecting elements of a list based on the position of two elements? Suppose I have the list below and I would like to select all the elements between 2 and 7, including them (note: not greater than/smaller than, but the elements that come after 2 and before 7 in the list):
scala> val l = List(1, 14, 2, 17, 35, 9, 12, 7, 9, 40)
l: List[Int] = List(1, 14, 2, 17, 35, 9, 12, 7, 9, 40)
scala> def someMethod(l: List[Int], from: Int, to: Int) : List[Int] = {
| // some code here
| }
someMethod: (l: List[Int], from: Int, to: Int)List[Int]
scala> someMethod(l, 2, 7)
res0: List[Int] = List(2, 17, 35, 9, 12, 7)
Expected output:
For lists that don't contain 2 and/or 7: an empty list
Input: (1, 2, 2, 2, 3, 4, 7, 8); Output: (2, 2, 2, 3, 4, 7)
Input: (1, 2, 3, 4, 7, 7, 7, 8); Output: (2, 3, 4, 7)
Input: (1, 2, 3, 4, 7, 1, 2, 3, 5, 7, 8); Output: ((2, 3, 4, 7), (2, 3, 5, 7))
Too bad that the regex-engines work only with strings, not with general lists - would be really nice if you could find all matches for something like L.*?R with two arbitrary delimiters L and R. Since it doesn't work with regex, you have to build a little automaton yourself. Here is one way to do it:
#annotation.tailrec
def findDelimitedSlices[A](
xs: List[A],
l: A,
r: A,
revAcc: List[List[A]] = Nil
): List[List[A]] = {
xs match {
case h :: t => if (h == l) {
val idx = xs.indexOf(r)
if (idx >= 0) {
val (s, rest) = xs.splitAt(idx + 1)
findDelimitedSlices(rest, l, r, s :: revAcc)
} else {
revAcc.reverse
}
} else {
findDelimitedSlices(t, l, r, revAcc)
}
case Nil => revAcc.reverse
}
}
Input:
for (example <- List(
List(1, 2, 2, 2, 3, 4, 7, 8),
List(1, 2, 3, 4, 7, 7, 7, 8),
List(1, 2, 3, 4, 7, 1, 2, 3, 5, 7, 8)
)) {
println(findDelimitedSlices(example, 2, 7))
}
Output:
List(List(2, 2, 2, 3, 4, 7))
List(List(2, 3, 4, 7))
List(List(2, 3, 4, 7), List(2, 3, 5, 7))
You're looking for slice:
# l.slice(2, 7)
res1: List[Int] = List(2, 17, 35, 9, 12)
# l.slice(2, 8)
res2: List[Int] = List(2, 17, 35, 9, 12, 7)
I have dataframe like this
+---------+--------------------+----------------------------+
| Name| rem1| quota |
+---------+--------------------+----------------------------+
|Customer_3|[258, 259, 260, 2...|[1, 2, 3, 4, 5, 6, 7,..500]|
|Customer_4|[18, 19, 20, 27, ...|[1, 2, 3, 4, 5, 6, 7,..500]|
|Customer_5|[16, 17, 51, 52, ...|[1, 2, 3, 4, 5, 6, 7,..500]|
|Customer_6|[6, 7, 8, 9, 10, ...|[1, 2, 3, 4, 5, 6, 7,..500]|
|Customer_7|[0, 30, 31, 32, 3...|[1, 2, 3, 4, 5, 6, 7,..500]|
I would like to remove list value in rem1 from quota and create as one new column. I have tried.
val dfleft = dfpci_remove2.withColumn("left",$"quota".filter($"rem1"))
<console>:123: error: value filter is not a member of org.apache.spark.sql.ColumnName
Please advise.
You can use a filter in a column in such way, you can write an udf as below
val filterList = udf((a: Seq[Int], b: Seq[Int]) => a diff b)
df.withColumn("left", filterList($"rem1", $"quota") )
This should give you the expected result.
Hope this helps!
I am trying to get the average from a mapped list of ints, then return that value to the user when requested.
Here is my current code with problems, what am I doing wrong? I have included my functionality to find the last element of a tail, that works.
// *******************************************************************************************************************
// application logic
// read data from file
val mapdata = readFile("data.txt")
// *******************************************************************************************************************
// UTILITY FUNCTIONS
//GETS THE DATA FROM THE DATA.TXT
def readFile(filename: String): Map[String, List[Int]] = {
processInput(Source.fromFile(filename).getLines)
}
def processInput(lines: Iterator[String]): Map[String, List[Int]] = {
Try {
lines.foldLeft(Map[String, List[Int]]()) { (acc, line) =>
val splitline = line.split(",").map(_.trim).toList
acc.updated(splitline.head, splitline.tail.map(_.toInt))
}
}.getOrElse {
println("Sorry, an exception happened.")
Map()
}
}
//functionality to find the last tail element
def findLast(list:List[Int]):Int = {
if(list.tail == Nil)
list.head
else
findLast(list.tail)
}
//Function to find the average
def average(list:List[Int]):Double =
list.foldLeft(0.0)(_+_) / list.foldLeft(0)((r,c)=>r+1)
//Show last element in the list, most current WORKS
def currentStockLevel (stock: String): (String, Int) = {
(stock, mapdata.get (stock).map(findLast(_)).getOrElse(0))
}
//Show last element in the list, most current DOES NOT WORK
def averageStockLevel (stock: String): (String, Int) = {
(stock, mapdata.get (stock).map(average(_)).getOrElse(0))
}
my txt file
SK1, 9, 7, 2, 0, 7, 3, 7, 9, 1, 2, 8, 1, 9, 6, 5, 3, 2, 2, 7, 2, 8, 5, 4, 5, 1, 6, 5, 2, 4, 1
SK2, 0, 7, 6, 3, 3, 3, 1, 6, 9, 2, 9, 7, 8, 7, 3, 6, 3, 5, 5, 2, 9, 7, 3, 4, 6, 3, 4, 3, 4, 1
SK4, 2, 9, 5, 7, 0, 8, 6, 6, 7, 9, 0, 1, 3, 1, 6, 0, 0, 1, 3, 8, 5, 4, 0, 9, 7, 1, 4, 5, 2, 8
SK5, 2, 6, 8, 0, 3, 5, 5, 2, 5, 9, 4, 5, 3, 5, 7, 8, 8, 2, 5, 9, 3, 8, 6, 7, 8, 7, 4, 1, 2, 3
The error that I am getting is that expression of type AnyVal does not conform to type Int
Your averageStockLevel function returns the average value as an Int (the return type is (String, Int)) whereas the calculation that is done in average returns a Double.
So you either need to convert the calculated Double to an Int within averageStockLevel (e.g. by doing average(_).toInt), or you can change the return type of averageStockLevel to (String, Double). The latter variant is obviously the better one since you don't loose the precision of your average value.
def averageStockLevel (stock: String): (String, Double) = {
(stock, mapdata.get(stock).map(average).getOrElse(0.0))
}
This works, but whether or not it's a good idea to return 0.0 in case of a missing key is for you to decide. Another possibility is to omit the getOrElse part and return an Option[(String,Double)].
Apart from that, your code is quite complex. findLast and average can be defined much easier (it's not really worth it to create an own function for finding the last element, but for the sake of completeness...):
// will throw an exception for empty lists, but so does your current code
def findLast(list:List[Int]) = list.last
def average(list:List[Int]): Double = list.sum.toDouble / list.size
Another idea is to replace List by Vector. For operations such as .size and .last, List needs linear time whereas Vector basically takes constant time.
I want to iterate over a scala list in an incremental way, i.e. the first pass should yield the head, the second the first 2 elements, the next the first 3, etc...
I can code this myself as a recursive function, but does a pre-existing function exist for this in the standard library?
You can use the .inits method to get there, albeit there may be performance issues for a large list (I haven't played around with making this lazy):
scala> val data = List(0,1,2,3,4)
data: List[Int] = List(0, 1, 2, 3, 4)
scala> data.inits.toList.reverse.flatten
res2: List[Int] = List(0, 0, 1, 0, 1, 2, 0, 1, 2, 3, 0, 1, 2, 3, 4)
You can use the take like so:
scala> val myList = 1 to 10 toList
myList: List[Int] = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
scala> for(cnt <- myList.indices) yield myList.take(cnt+1)
res1: scala.collection.immutable.IndexedSeq[List[Int]] = Vector(List(1), List(1, 2), List(1, 2, 3), List(1, 2, 3, 4), List(1, 2, 3, 4, 5), List(1, 2, 3, 4, 5, 6), List(1, 2, 3, 4, 5, 6, 7), List(1, 2, 3, 4, 5, 6, 7, 8), List(1, 2, 3, 4, 5, 6, 7, 8, 9), List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10))
OK, since I've whined enough, here's an iterator version that tries reasonably hard to not waste space or compute more than is needed at at one point:
class stini[A](xs: List[A]) extends Iterator[List[A]] {
var ys: List[A] = Nil
var remaining = xs
def hasNext = remaining.nonEmpty
def next = {
val e = remaining.head
remaining = remaining.tail
ys = e :: ys
ys.reverse
}
}
val it = new stini(List(1, 2, 3, 4))
it.toList
//> List[List[Int]] =
// List(List(1), List(1, 2), List(1, 2, 3), List(1, 2, 3, 4))
Try: for((x, i) <- l.view.zipWithIndex) println(l.take(i + 1))
if you need something side-effected (I just did println to give you an example)
I'm trying to get List(0,1,2,...n)
Is there a cleaner/better way than:
scala> List(0 to 9)
res0: List[scala.collection.immutable.Range.Inclusive] = List(Range(0, 1, 2, 3, 4,
5, 6, 7, 8, 9))
scala> List(0 to 9).flatten
res1: List[Int] = List(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
The best way might be:
(0 to 9).toList
scala> List.range(0, 10)
res0: List[Int] = List(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
Also
List(0 to 9: _*)
I suspect though that List.range is the most efficient one.