Value * is not a member of AnyVal - scala

This is a fold that I wrote and I get this error:
Error:(26, 42) value * is not a member of AnyVal
(candE.intersect(candR), massE * massR)
^
allAssignmentsTable is a List[Map[Set[Candidate[A]],Double]]
val allAssignmentsTable = hypothesis.map(h => {
allAssignments.map(copySet => {
if(h.getAssignment.keySet.contains(copySet))
(copySet -> h.getAssignment(copySet))
else
(copySet -> 0.0)
}).toMap
})
val aggregated = allAssignmentsTable.foldLeft(initialFold) { (res,element) =>
val allIntersects = element.map {
case (candE, massE) =>
res.map {
case (candR, massR) => candE.intersect(candR), massE * massR
}.toList
}.toList.flatten
val normalizer = allIntersects.groupBy(_._1).filter(_._1.size == 0).map {
case(key, value) => value.foldLeft(0.0)((e,i) => i._2 + e)
}.head
allIntersects.groupBy(_._1).map {
case(key, value) => key -> value.foldLeft(0.0)((e,i) => i._2 + e)
}
}
if I do this: case(candE, massE:Double) then I won't get an error but I will get exception in match.

The problem that you get here:
val aggregated = allAssignmentsTable.foldLeft(initialFold) { (res,element) =>
val allIntersects = element.map {
case (candE, massE) =>
res.map {
case (candR, massR) => candE.intersect(candR), massE * massR
}.toList
}.toList.flatten
is most probably arising from the previous code block:
val allAssignmentsTable = hypothesis.map(h => {
allAssignments.map(copySet => {
if(h.getAssignment.keySet.contains(copySet))
(copySet -> h.getAssignment(copySet))
else
(copySet -> 0.0)
}).toMap
})
My hypothesis is that h.getAssignment(copySet) returns something else instead of Double (which seems to be confirmed by the error message quoted in the OP - (26, 42)etc, neither of these two values look like it is a Double. Therefore, allAssignmentsTable undercover is probably not List[Map[Set[Candidate[A]],Double]] but something else e.g. it has Any instead of Double, therefore operator * cannot be applied.

Related

Exception handling in Spark Scala UDF

def parse_values(value: String) = {
val values = value.split(",").map(_.trim)
values.foldLeft(Array[(Int, Double)]()) {
case (acc, present) =>
val Array(k, v) = present.split(",")(0).split(":")
acc :+ (k.trim.toInt, v.trim.toDouble)
}
I am currently using the above UDF to parse a column of string into an array of keys and values.
"50:63.25,100:58.38" to [[50,63.2], [100,58.38]].
In some cases, the string is "\N" and I am unable to parse the column value.
If the string is "\N" then I should return an empty array. Can anyone help me to handle this exception or help me adding a new case? I am new to spark-scala.
Error: scala.MatchError: [Ljava.lang.String;#497cb6a9 (of class [Ljava.lang.String;)
You need to check that the resulting Array has two elements. You need a pattern matching like this to avoid that parse error:
def parse_values(value: String) = {
val values = value.split(",").map(_.trim)
values.foldLeft(Array[(Int, Double)]()) {
case (acc, present) =>
val Array(k, v) = {
present.split(",")(0).split(":") match {
case Array(_) => Array("0", "0.0")
case arr => arr
}
}
acc :+ (k.trim.toInt, v.trim.toDouble)
}
}

Scala functional programming dry run

Could you please help me in understanding the following method:
def extractGlobalID(custDimIndex :Int)(gaData:DataFrame) : DataFrame = {
val getGlobId = udf[String,Seq[GenericRowWithSchema]](genArr => {
val globId: List[String] =
genArr.toList
.filter(_(0) == custDimIndex)
.map(custDim => custDim(1).toString)
globId match {
case Nil => ""
case x :: _ => x
}
})
gaData.withColumn("globalId", getGlobId('customDimensions))
}
The method applies an UDF to to dataframe. The UDF seems intended to extract a single ID from column of type array<struct>, where the first element of the struct is an index, the second one an ID.
You could rewrite the code to be more readable:
def extractGlobalID(custDimIndex :Int)(gaData:DataFrame) : DataFrame = {
val getGlobId = udf((genArr : Seq[Row]) => {
genArr
.find(_(0) == custDimIndex)
.map(_(1).toString)
.getOrElse("")
})
gaData.withColumn("globalId", getGlobId('customDimensions))
}
or even shorter with collectFirst:
def extractGlobalID(custDimIndex :Int)(gaData:DataFrame) : DataFrame = {
val getGlobId = udf((genArr : Seq[Row]) => {
genArr
.collectFirst{case r if(r.getInt(0)==custDimIndex) => r.getString(1)}
.getOrElse("")
})
gaData.withColumn("globalId", getGlobId('customDimensions))
}

scopt3 sample script does not compile

I'm trying to use scopt3 in my project but I get compilation errors even for the sample code on scopt3 Github page:
val parser = new scopt.OptionParser[Config]("scopt") {
head("scopt", "3.x")
opt[Int]('f', "foo") action { (x, c) =>
c.copy(foo = x) } text("foo is an integer property")
opt[File]('o', "out") required() valueName("<file>") action { (x, c) =>
c.copy(out = x) } text("out is a required file property")
opt[(String, Int)]("max") action { case ((k, v), c) =>
c.copy(libName = k, maxCount = v) } validate { x =>
if (x._2 > 0) success else failure("Value <max> must be >0")
} keyValueName("<libname>", "<max>") text("maximum count for <libname>")
opt[Seq[File]]('j', "jars") valueName("<jar1>,<jar2>...") action { (x,c) =>
c.copy(jars = x) } text("jars to include")
opt[Map[String,String]]("kwargs") valueName("k1=v1,k2=v2...") action { (x, c) =>
c.copy(kwargs = x) } text("other arguments")
opt[Unit]("verbose") action { (_, c) =>
c.copy(verbose = true) } text("verbose is a flag")
opt[Unit]("debug") hidden() action { (_, c) =>
c.copy(debug = true) } text("this option is hidden in the usage text")
note("some notes.\n")
help("help") text("prints this usage text")
arg[File]("<file>...") unbounded() optional() action { (x, c) =>
c.copy(files = c.files :+ x) } text("optional unbounded args")
cmd("update") action { (_, c) =>
c.copy(mode = "update") } text("update is a command.") children(
opt[Unit]("not-keepalive") abbr("nk") action { (_, c) =>
c.copy(keepalive = false) } text("disable keepalive"),
opt[Boolean]("xyz") action { (x, c) =>
c.copy(xyz = x) } text("xyz is a boolean property"),
checkConfig { c =>
if (c.keepalive && c.xyz) failure("xyz cannot keep alive") else success }
)
}
errors are:
1) not found type Config
I thougth it was com.typesafe.config.Config, but when I import I get "velue copy is not a member of com.typesafe.config.Config". Where does Config come from?
2) not found value foo
All arguments to .copy() method are marked as "not found values" (I suppose is due to the previous error on Config)
I'm on scala 2.11.6 / SBT 0.13.8
Any help?
scopt has nothing to do with typesafe-config at all.
If you carefully read the README, you will notice that Config is defined right before the code you paste here:
case class Config(foo: Int = -1, out: File = new File("."), xyz: Boolean = false,
libName: String = "", maxCount: Int = -1, verbose: Boolean = false, debug: Boolean = false,
mode: String = "", files: Seq[File] = Seq(), keepalive: Boolean = false,
jars: Seq[File] = Seq(), kwargs: Map[String,String] = Map())
It is merely an example of an object that holds all parameters of a program.
The idea is you parametrise OptionParser with any configuration object you wish. E.g.
case class Foo(name: String)
val parser = new scopt.OptionParser[Foo]("foo") {
opt[String]('n', "name") action { (n, foo) => foo.copy(name = n) }
}
val res = parser.parse(args, Foo("default"))

Tuple seen as Product, compiler rejects reference to element

Constructing phoneVector:
val phoneVector = (
for (i <- 1 until 20) yield {
val p = killNS(r.get("Phone %d - Value" format(i)))
val t = killNS(r.get("Phone %d - Type" format(i)))
if (p == None) None
else
if (t == None) (p,"Main") else (p,t)
}
).filter(_ != None)
Consider this very simple snippet:
for (pTuple <- phoneVector) {
println(pTuple.getClass.getName)
println(pTuple)
//val pKey = pTuple._1.replaceAll("[^\\d]","")
associate() // stub prints "associate"
}
When I run it, I see output like this:
scala.Tuple2
((609) 954-3815,Mobile)
associate
When I uncomment the line with replaceAll(), compile fails:
....scala:57: value _1 is not a member of Product with Serializable
[error] val pKey = pTuple._1.replaceAll("[^\\d]","")
[error] ^
Why does it not recognize pTuple as a Tuple2 and treat it only as Product
OK, this compiles and produces the desired result. But it's too verbose. Can someone please demonstrate a more concise solution for dealing with this typesafe stuff?
for (pTuple <- phoneVector) {
println(pTuple.getClass.getName)
println(pTuple)
val pPhone = pTuple match {
case t:Tuple2[_,_] => t._1
case _ => None
}
val pKey = pPhone match {
case s:String => s.replaceAll("[^\\d]","")
case _ => None
}
println(pKey)
associate()
}
You can do:
for (pTuple <- phoneVector) {
val pPhone = pTuple match {
case (key, value) => key
case _ => None
}
val pKey = pPhone match {
case s:String => s.replaceAll("[^\\d]","")
case _ => None
}
println(pKey)
associate()
}
Or simply phoneVector.map(_._1.replaceAll("[^\\d]",""))
By changing the construction of phoneVector, as wrick's question implied, I've been able to eliminate the match/case stuff because Tuple is assured. Not thrilled by it, but Change is Hard, and Scala seems cool.
Now, it's still possible to slip a None value into either of the Tuple values. My match/case does not check for that, and I suspect that could lead to a runtime error in the replaceAll call. How is that allowed?
def killNS (s:Option[_]) = {
(s match {
case _:Some[_] => s.get
case _ => None
}) match {
case None => None
case "" => None
case s => s
}
}
val phoneVector = (
for (i <- 1 until 20) yield {
val p = killNS(r.get("Phone %d - Value" format(i)))
val t = killNS(r.get("Phone %d - Type" format(i)))
if (t == None) (p,"Main") else (p,t)
}
).filter(_._1 != None)
println(phoneVector)
println(name)
println
// Create the Neo4j nodes:
for (pTuple <- phoneVector) {
val pPhone = pTuple._1 match { case p:String => p }
val pType = pTuple._2
val pKey = pPhone.replaceAll(",.*","").replaceAll("[^\\d]","")
associate(Map("target"->Map("label"->"Phone","key"->pKey,
"dial"->pPhone),
"relation"->Map("label"->"IS_AT","key"->pType),
"source"->Map("label"->"Person","name"->name)
)
)
}
}

mongodb casbah and list handling

I am having problems writing this function, which takes a string and returns a list of strings associated to it.
(I'm expecting entries like {_id: ...., hash: "abcde", n: ["a","b","ijojoij"]} in mongodb)
def findByHash(hash: Hash) = {
val dbobj = mongoColl.findOne(MongoDBObject("hash" -> hash.hashStr))
val n = dbobj match {
case Some(doc: com.mongodb.casbah.Imports.DBObject) => {
doc("n") match {
case Some(n: com.mongodb.casbah.Imports.DBObject) => {
Some(List[String]() ++ n map { x => x.asInstanceOf[String] })
}
case _ => {
None // hash match but no n in object
}
}
}
case _ => {
None // no hash match
}
}
n
}
Is there anything wrong with the code? Do you know how to correct it?
doc("n") returns AnyRef, so you should explicitly cast it to BasicDBList.
val n = doc("n").asInstanceOf[BasicDBList]
Some(List[String]() ++ n map { x => x.asInstanceOf[String] })