I want to find minimum value over RDD triples. This is how I approach this task (saving to file is done for validation).
evalsRDD.coalesce(1)
.map(e => "%.3f\t%d\t%.3f".format(e._1, e._2, e._3))
.saveAsTextFile("data/streaming/test")
val minRMSE = evalsRDD.min()(new Ordering[(Any, Any, Double)]() {
override def compare(x: (Any, Any, Double), y: (Any, Any, Double)): Double =
Ordering[Double].compare(x._3, y._3) })
It says that method compare has incompatible type. How to solve this issue?
Your compare method must return an Int, not a Double, to override the method defined in Ordering.
If you change it do to so, it works:
val minRMSE = evalsRDD.min()(new Ordering[(Any, Any, Double)]() {
override def compare(x: (Any, Any, Double), y: (Any, Any, Double)): Int =
Ordering[Double].compare(x._3, y._3)
})
Related
I need to call a sort method of a library that takes an implicit Ordering parameter by using implicitly like this:
class OrderedRDDFunctions[K : Ordering : ClassTag,
V: ClassTag] (self: RDD[P]) {
private val ordering = implicitly[Ordering[K]]
def sort() = {
// uses ordering value
}
}
Now, I need to call this function twice in a loop, with different Orderings of same type like below.
var A: RDD[(Int, Int, Int)] = ...
var C: RDD[(Int, Int, Int)] = ...
while(...) {
implicit val ijOrdering:Ordering[(Int, Int, Int)] = new Ordering[(Int, Int, Int)] {
override def compare(a: (Int, Int, Int), b: (Int, Int, Int)) = {
val c = a._1.compare(b._1)
if(c != 0) c
else a._2.compare(b._2)
}
}
A.sort() // should use ijOrdering above
implicit val kjOrdering:Ordering[(Int, Int, Int)] = new Ordering[(Int, Int, Int)] {
override def compare(a: (Int, Int, Int), b: (Int, Int, Int)) = {
val c = a._3.compare(b._3)
if(c != 0) c
else a._2.compare(b._2)
}
}
C.sort() // should use kjOrdering above
}
There are two different implicit Ordering instances to be used in sort() method. But this gives me a compile error. How can I state different implicit ordering in this setup? Note that I cannot change the library method.
You can use blocks to limit the scope of implicits. A simple example:
object Blocks extends App {
def meth()(implicit e: Int) = e * 2
locally {
implicit val k = 21
println(meth()) // finds k and prints 42
}
locally {
implicit val j = 11
println(meth()) // finds j and prints 22
}
}
This does not work if there are conflicting implicits outside of the block.
Note that locally { ..stmts.. } is generally equivalent to { ..stmts.. }, tho I prefer it for readability's sake. You can read more on what's that here
I have a function that calculate the tuple values only when the input is a tuple of four integers.
def add(v:Any) = {
if (v.isInstanceOf[(Int, Int, Int, Int)]) {
val v2 = v.asInstanceOf[(Int, Int, Int, Int)]
println(v2._1 + v2._2 + v2._3 + v2._4)
} else {
println("NOP")
}
}
object Main extends App {
add((1,1,1,1))
add((1,2))
}
Main.main(args)
It's working, but I got a warning of "non-variable type argument ... is unchekced" warning.
warning: non-variable type argument Int in type (Int, Int, Int, Int) is
unchecked since it is eliminated by erasure
if (v.isInstanceOf[(Int, Int, Int, Int)]) {
Why this error, and what might be the best way to remove this warning message?
If you actually need to check that the argument is a tuple of four Ints, the correct way to do this is to check each component:
def add(v: Any) = v match {
case (i1: Int, i2: Int, i3: Int, i4: Int) =>
println(i1 + i2 + i3 + i4)
case _ =>
println("NOP")
}
This is caused by type erasure in compile time, you can resolve it by TypeTag:
import scala.reflect.runtime.universe._
def add[T](v:T)(implicit t: TypeTag[T]) = {
if ( t.tpe =:= typeOf[(Int, Int, Int, Int)]) {
val v2 = v.asInstanceOf[(Int, Int, Int, Int)]
println(v2._1 + v2._2 + v2._3 + v2._4)
} else {
println("NOP")
}
}
You can replace instanceOfs with pattern matching and suppress the warning with #unchecked
def add(v: Any) = v match {
case t: (Int, Int, Int, Int) #unchecked =>
println(t._1 + t._2 + t._3 + t._4)
case _ =>
println("NOP")
}
if you pass a Tuple4 that is not (Int, Int, Int, Int) you will get a ClassCastException
Error clearly says that generic types of tuple will be removed due to type erasure and hence compiler can't assure that this will work in runtime, it will only see if Tuple4 was passed, not what it contains.
The solution I presented would cause you trouble if it is possible that the function would be called with other than (Int, Int, Int, Int) Tuple4, and then you should proceed with TypeTags, otherways it just looks so much cleaner and doesn't need reflection.
I want to define an auxiliary constructor which should compute some value and use it in multiple parameters passed to a primary constructor. A natural way to express this would be to define local vals before calling the primary constructor, but this is not allowed by the language:
object Computed {
def veryComplexComputation(x: Double) = math.sqrt(x) // imagine there is some much more complex computation here
}
class Computed(x: Double, y: Double, z: Double) {
def this(x: Double, y: Double) = {
val derivedFromX = Computed.veryComplexComputation(x)
val derivedFromY = derivedFromX * Computed.veryComplexComputation(y)
this(x, derivedFromX, derivedFromY)
}
}
An alternative is to repeat the computation, but with very complex computations this can be a problem (and it also leads to a code repetition):
class Computed(x: Double, y: Double, z: Double) {
def this(x: Double, y: Double) = {
this(
x,
Computed.veryComplexComputation(x),
Computed.veryComplexComputation(x) * Computed.veryComplexComputation(y)
)
}
}
The best trick I know is to use a third, private constructor taking a tuple of the parameters for the primary constructor, together with a helper function in the companion object:
object Computed {
def veryComplexComputation(x: Double) = math.sqrt(x) // imagine there is some much more complex computation here
private def computeArgs(x: Double, y: Double): (Double, Double, Double) = {
val derivedFromX = veryComplexComputation(x)
val derivedFromY = derivedFromX * veryComplexComputation(y)
(x, derivedFromX, derivedFromY)
}
}
class Computed(x: Double, y: Double, z: Double) {
private def this(xyz: (Double, Double, Double)) =
this(xyz._1, xyz._2, xyz._3)
def this(x: Double, y: Double) =
this(Computed.computeArgs(x, y))
}
I think the best way is to add a suitable factory method to the companion object:
object Computed {
def veryComplexComputation(x: Double) = math.sqrt(x) // can be private, if you wish
def apply(x: Double, y: Double): Computed = {
val derivedFromX = veryComplexComputation(x)
val derivedFromY = derivedFromX * veryComplexComputation(y)
new Computed(x, derivedFromX, derivedFromY)
}
}
then you can instantiate your Computed type the same way as for case classes, foregoing the use of "new":
scala> val comp = Computed(4.0, 9.0)
comp: Computed = Computed#4de4e24f
(You might also want to consider making Computed a case class anyway - particularly if is entirely, or even mostly, used as a container for data)
One can (ab)use multiple parameter lists with default values, like this:
class Computed(x: Double, y: Double, z: Double) {
def this(x: Double, y: Double)(
derivedFromX: Double = Computed.veryComplexComputation(x)
)(
derivedFromY: Double = derivedFromX * Computed.veryComplexComputation(y)
) = {
this(x, derivedFromX, derivedFromY)
}
}
This exploits the fact each parameter list can access parameters from previous lists. Just do not forget to provide the empty parameter lists when calling the constructor:
new Computed(1, 2)()()
One can chain several auxiliary constructors, each of them performing one computation, using marker traits to distinguish between constructors to avoid overloading ambiguity.
object Computed {
def veryComplexComputation(x: Double) = math.sqrt(x) // imagine there is some much more complex computation here
private trait DerivedFromX
private object DerivedFromX extends DerivedFromX
}
class Computed(x: Double, y: Double, z: Double) {
private def this(name: Computed.DerivedFromX, x: Double, derivedFromX: Double, y: Double) = {
this(x, derivedFromX, derivedFromX * Computed.veryComplexComputation(y))
}
def this(x: Double, y: Double) = this(Computed.DerivedFromX, x, Computed.veryComplexComputation(x), y)
}
I keep getting error: type mismatch: found Uint required Double
When I put a println function in the sqrt method. While I appreciate its part of Scala preventing side effects, how do print values in functions so I can make sense of my programs? Is there a "clean" way to print values in a function that needs an explicit return type (like the recursive function sqrt)?
Code here:
object Newton {
def threshold(guess: Double, x: Double) : Boolean =
if (Math.abs(guess * guess -x) < (0.01/100 * x)) true else false
def improve(guess: Double, x: Double) : Double =
(guess + x/guess) / 2.0
def sqrt(guess: Double, x: Double,
threshold: (Double, Double) => Boolean,
improve: (Double, Double) => Double ): Double =
println("current guess:", guess)
if(threshold(guess,x))
return guess
else
return sqrt(improve(guess, x), x, threshold, improve)
def main(args: Array[String]): Unit = {
println("Sqrt of Two:", sqrt(1,1.0e-20,threshold, improve))
}
}
You are missing curly braces.
def sqrt(guess: Double, x: Double,
threshold: (Double, Double) => Boolean,
improve: (Double, Double) => Double ): Double = { // Add curly braces
println("current guess:", guess)
if(threshold(guess,x))
return guess
else
return sqrt(improve(guess, x), x, threshold, improve)
} // Add curly braces
I have the following:
def calc(dir: File): Option[Group] = {
calcModel(dir) match {
case Some(model: Model) => calcUI(model, centerpane.getWidth, centerpane.getHeight, 5, MAX_LEVEL)
case None => None
}
}
def calcUI(model: Model, availableWidth: Double, availableHeight: Double, maxLevel: Int): Option[Group] = {
}
def calcUI(model: Model, posX: Double, posY: Double, availableWidth: Double, availableHeight: Double, horizontally: Boolean, startLevel: Int, maxLevel: Int): Option[Group] = {
}
protected def calcUI(node: Node, posX: Double, posY: Double, availableWidth: Double, availableHeight: Double, horizontally: Boolean, level: Int, maxLevel: Int): Group = {
}
def calcModel(dir: File): Option[Model] = {
}
(Remark: Model does NOT derive from Node)
and get the following compiler error message:
Error:(88, 27) overloaded method value calcUI with alternatives:
(node: org.rob.spaceview.modell.Node,posX: Double,posY: Double,availableWidth: Double,availableHeight: Double,horizontally: Boolean,level: Int,maxLevel: Int)javafx.scene.Group <and>
(model: org.rob.spaceview.modell.Model,posX: Double,posY: Double,availableWidth: Double,availableHeight: Double,horizontally: Boolean,startLevel: Int,maxLevel: Int)Option[javafx.scene.Group] <and>
(model: org.rob.spaceview.modell.Model,availableWidth: Double,availableHeight: Double,maxLevel: Int)Option[javafx.scene.Group]
cannot be applied to (org.rob.spaceview.modell.Model, Double, Double, Int, Int)
case Some(model) => calcUI(model, centerpane.getWidth, centerpane.getHeight, 5, MAX_LEVEL)
^
I do not get it. All calcUI functions are different by parameters.
I know this error and most of the time I get it and fix it. But here, no clue.
Hopefully somebody enlights me. Thanks.
Actually, the given parameters in the call
case Some(model) => calcUI(model, centerpane.getWidth, centerpane.getHeight, 5, MAX_LEVEL)
don't correspond to any defined calcUI method. For example for the following definition
def calcUI(model: Model, availableWidth: Double, availableHeight: Double, maxLevel: Int): Option[Group]
you have one more argument than needed.