I am learning Scala at the moment
<pre>
scala> val sample = similarities.filter(m => {
| val movies = m._1
| (movieNames(movies._1).contains("Star Wars (1977)"))
| })
</pre>
sample: org.apache.spark.rdd.RDD[((Int, Int), Double)] = FilteredRDD[25] at filter at :36
The sample compiled just fine
But when I tried to call sample again in the next command
<pre>
scala> val result = sample.map(v => {
| val m1 = v._1._1
| val m2 = v._1._2
| val correl = v._2._1
| //val rcorr = v._2._2
| // val cos = v._2._3
| //val j = v._2._4
| (movieNames(m1), movieNames(m2), correl)
| })
<console>:41: error: value _1 is not a member of Double
val correl = v._2._1
</pre>
Can someone please help me.Thanks in advance
Given the amount of indexing on the composed tuple, consider wrapping ((Int, Int), Double) onto a case class and defining an implicit on it as follows,
case class Movie(m1: Int, m2: Int, correl: Double)
implicit def RichMovie(v: ((Int,Int),Double) ) = Movie(v._1._1, v._1._2, v._2)
Thus, given an instance of a composed tuple
scala> val m = ( (1,2), 3.5)
m: ((Int, Int), Double) = ((1,2),3.5)
we can access its members as follows,
scala> m.m1
res0: Int = 1
scala> m.m2
res1: Int = 2
scala> m.correl
res2: Double = 3.5
val correl = v._2._1
should be just
val correl = v._2
because it is part of the second element in the tuple ((Int, Int), Double)
Related
In the following example, I am using view bound and upper bound. The code is exactly the same except for the difference in signs of <% and <:. What is the difference between view bound and upper bound?
scala> class Person[T <% Ordered[T]](val fn:T, val ln:T) {
| def greater = if (fn > ln) fn else ln
| }
defined class Person
scala> val p1 = new Person("manu", "chadha")
p1: Person[String] = Person#31b91435
scala> val p2 = new Person("anita", "chadha")
p2: Person[String] = Person#61e0f9b9
scala> p1.greater
res3: String = manu
scala> p2.greater
res4: String = chadha
scala> class Person2[T <: Ordered[T]](val fn:T, val ln:T) {
| def greater = if (fn > ln) fn else ln
| }
defined class Person2
scala> val p3 = new Person("manu", "chadha")
p3: Person[String] = Person#7c50709a
scala> val p4 = new Person("anita", "chadha")
p4: Person[String] = Person#1d686622
scala> p3.greater
res5: String = manu
scala> p4.greater
res6: String = chadha
scala>
I use scala f string interpolator as follows:
def format(id: Int) = f"A$id%04d"
format(21) // A0021
However, I would like to be able to define a length once and for all (before fixed to 4), and get a function that it is going to format the string with that length.
So, instead of having
def format(length: Int, id: Int) = ???
f(5, 21) // A00021
I would like to have this:
def format(length: Int)(id: Int) = ???
val f = format(5)
f(21) // A00021
How can I implement this using scala f interpolator or other?
Update
I was not looking for such a solution involving the compiler at runtime, but I appreciate som-snytt's answer. Here there is a working solution based on his answer:
import scala.tools.reflect._,scala.reflect.runtime._,universe._
def defFormat(length: Int): Int => String = {
val code = raw"""(i: Int) => f"A$$i%0${length}d""""
tb.eval(tb.parse(code)).asInstanceOf[Int => String]
}
val format = defFormat(length = 5)
format(21)
scala> def format(n: Int)(i: Int) =
| f"A%%0${n}d" format i
format: (n: Int)(i: Int)String
scala> format(5) _
res0: Int => String = <function1>
scala> .apply(21)
res1: String = A00021
Edit:
scala> import scala.tools.reflect._,scala.reflect.runtime._,universe._
import scala.tools.reflect._
import scala.reflect.runtime._
import universe._
scala> val tb = currentMirror.mkToolBox()
tb: scala.tools.reflect.ToolBox[reflect.runtime.universe.type] = scala.tools.reflect.ToolBoxFactory$ToolBoxImpl#2d10e0b1
scala> def f(n: Int)(i: Int): String = {
| val code = raw"""f"A$${$i}%0${n}d""""
| tb.eval(tb.parse(code)).asInstanceOf[String]
| }
f: (n: Int)(i: Int)String
scala> val g = f(5) _
g: Int => String = <function1>
scala> g(21)
res9: String = A00021
That doesn't actually help much. You really want to
scala> tb.typecheck(tb.parse(code))
scala.tools.reflect.ToolBoxError: reflective typecheck has failed: illegal conversion character 'k'
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal$$anonfun$typecheck$1.apply(ToolBoxFactory.scala:178)
which throws if the format is bad.
scala> val code = raw"""(i: Int) => f"A$${i}%k0${10}d""""
code: String = (i: Int) => f"A${i}%k010d"
scala> tb.typecheck(tb.parse(code))
scala.tools.reflect.ToolBoxError: reflective typecheck has failed: illegal conversion character 'k'
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal$$anonfun$typecheck$1.apply(ToolBoxFactory.scala:178)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal$$anonfun$typecheck$1.apply(ToolBoxFactory.scala:170)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal$$anonfun$transformDuringTyper$1$$anonfun$11.apply(ToolBoxFactory.scala:148)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal$$anonfun$transformDuringTyper$1$$anonfun$11.apply(ToolBoxFactory.scala:148)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal$$anonfun$transformDuringTyper$1$$anonfun$9.apply(ToolBoxFactory.scala:138)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal$$anonfun$transformDuringTyper$1$$anonfun$9.apply(ToolBoxFactory.scala:138)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal$$anonfun$transformDuringTyper$1$$anonfun$withContext$1$1.apply(ToolBoxFactory.scala:139)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal$$anonfun$transformDuringTyper$1$$anonfun$withContext$1$1.apply(ToolBoxFactory.scala:139)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal$$anonfun$transformDuringTyper$1$$anonfun$7.apply(ToolBoxFactory.scala:137)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal$$anonfun$transformDuringTyper$1$$anonfun$7.apply(ToolBoxFactory.scala:137)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal$$anonfun$transformDuringTyper$1.apply(ToolBoxFactory.scala:148)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal$$anonfun$transformDuringTyper$1.apply(ToolBoxFactory.scala:121)
at scala.reflect.internal.Trees$class.wrappingIntoTerm(Trees.scala:1716)
at scala.reflect.internal.SymbolTable.wrappingIntoTerm(SymbolTable.scala:16)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal.withWrapping$1(ToolBoxFactory.scala:120)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal.transformDuringTyper(ToolBoxFactory.scala:121)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal.typecheck(ToolBoxFactory.scala:169)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$$anonfun$typecheck$2.apply(ToolBoxFactory.scala:375)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$$anonfun$typecheck$2.apply(ToolBoxFactory.scala:367)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$withCompilerApi$.liftedTree2$1(ToolBoxFactory.scala:355)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$withCompilerApi$.apply(ToolBoxFactory.scala:355)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl.typecheck(ToolBoxFactory.scala:367)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl.typecheck(ToolBoxFactory.scala:27)
... 32 elided
scala> val code = raw"""(i: Int) => f"A$${i}%0${10}d""""
code: String = (i: Int) => f"A${i}%010d"
scala> tb.typecheck(tb.parse(code))
res19: tb.u.Tree =
((i: Int) => ({
val arg$macro$9: Int = i;
new scala.collection.immutable.StringOps("A%010d").format(arg$macro$9)
}: String))
You can't do it using f because its whole point is to make sure it can check the format string for type errors, so the format string has to be static. f could support this scenario explicitly, but it doesn't.
You could make format a macro, but this seems like an overkill. Not to mention that it would have to be defined in a separate module, which looks very inconvenient for this scenario.
I have the following use case, in which I am iterating multiple times on the same collection, and every time I find a different item in that collection.
class Foo(policyToData: Map[String, MyClass]){
val milk: Option[MyClass] = policyToData.values.find(_.`type` == Milk)
val meat: Option[MyClass] = policyToData.values.find(_.`type` == Meat)
val bread: Option[MyClass] = policyToData.values.find(_.`type` == Bread)
val other: List[MyClass] = policyToData.values.filter(_.`type` == Other).toList
}
Is there a better way to do it? with one iteration?
If it's a large collection, folding into a map means you only build the collection of interest.
scala> case class C(name: String)
defined class C
scala> val cs = List(C("milk"),C("eggs"),C("meat"))
cs: List[C] = List(C(milk), C(eggs), C(meat))
scala> cs.foldLeft(Map.empty[String,C]) {
| case (m, c # C("milk" | "meat")) if !m.contains(c.name) => m + (c.name -> c)
| case (m, _) => m }
res5: scala.collection.immutable.Map[String,C] = Map(milk -> C(milk), meat -> C(meat))
then
scala> val milk = res5("milk")
milk: C = C(milk)
scala> val bread = res5.get("bread")
bread: Option[C] = None
The original groupBy solution was deleted because someone commented that it does extra work, but in fact it's a straightforward expression, if creating the intermediate Map of Lists is OK.
scala> cs.groupBy(_.name)
res0: scala.collection.immutable.Map[String,List[C]] = Map(meat -> List(C(meat)), eggs -> List(C(eggs)), milk -> List(C(milk)))
scala> res0.get("milk").map(_.head)
res1: Option[C] = Some(C(milk))
scala> res0.get("bread").map(_.head)
res2: Option[C] = None
or
scala> cs.filter { case C("milk" | "meat") => true case _ => false }.groupBy(_.name)
res4: scala.collection.immutable.Map[String,List[C]] = Map(meat -> List(C(meat)), milk -> List(C(milk)))
groupBy will do it:
val byType = list.groupBy(_.type).withDefaultValue(Nil)
val milk = byType(Milk).headOption
val other = byType(Other)
Etc ...
Is there a supported way to achieve a conversion of any numeric type to a double. E.g.
val i = 12345
val f = 1234.5F
val d = 1234.5D
val arr = Array[Any](i,f,d)
val anotherD = arr(0).asInstanceOf[Numeric].toDouble
Naturally the above code is not correct as given - since Numeric requires Type arguments.
scala> val i = 12345
i: Int = 12345
scala> val f = 1234.5F
f: Float = 1234.5
scala> val d = 1234.5D
d: Double = 1234.5
scala> val arr = Array[Any](i,f,d)
arr: Array[Any] = Array(12345, 1234.5, 1234.5)
scala> val anotherD = arr(0).asInstanceOf[Numeric].toDouble
<console>:11: error: type Numeric takes type parameters
val anotherD = arr(0).asInstanceOf[Numeric].toDouble
Now I realize the above may be achieved via match/case , along the following lines:
(a, e) match {
case (a : Double, e : Double) =>
Math.abs(a - e) <= CompareTol
case (a : Float, e : Float) =>
Math.abs(a - e) <= CompareTol
.. etc
But I was wondering if there were a means to more compactly express the operation. This code is within TEST classes and efficiency is not an important criterion. Specifically: reflection calls are OK. Thanks.
I assume you are on the JVM. The Number class does like what you want to achieve with the doubleValue method:
val arr = Array[Number](i,f,d)
val ds = arr.map(_.doubleValue())
This is horrible, and probably not efficient, but it works (on your example) :p
scala> import scala.language.reflectiveCalls
import scala.language.reflectiveCalls
scala> arr.map(_.asInstanceOf[{ def toDouble: Double }].toDouble)
res2: Array[Double] = Array(12345.0, 1234.5, 1234.5)
The preferred approach would be to use something similar to the commented out line below.
def main(args: Array[String]) {
// val (dbPropsFile, tsvFile, dbTable) = args
val dbPropsFile = args(0)
val tsvFile = args(1)
val dbTable = args(2)
However I am having a little quarrel with the compiler over it:
Error:(13, 9) constructor cannot be instantiated to expected type;
found : (T1, T2, T3)
required: Array[String]
val (dbPropsFile, tsvFile, dbTable) = args
^
So all told this should be an easy few points for someone out there.
Use
val Array(dbPropsFile, tsvFile, dbTable) = args
scala> val Array(a,b,c) = Array(1,2,3)
a: Int = 1
b: Int = 2
c: Int = 3
scala> a
res0: Int = 1