I would like to add two new operations to a Scala Enumeration to get the previous and the next value given a certain value if it exists. For example, I would like to write something like:
object Nums extends MyNewEnumerationType {
type Nums = Value
val One,Two,Three = Value
}
Nums.nextOf(One) // Some(Two)
Nums.prevOf(One) // None
My idea was to create a new class and add the methods in this way:
class PrevNextEnum extends Enumeration {
val prevOf = values.zip(None +: values.map{_.some}.toSeq).toMap
val nextOf = {
if (values.isEmpty) Map.empty
else values.zip(values.tail.map{_.some}.toSeq :+ None).toMap
}
}
The problem is that this doesn't work because when prevOf and nextOf are initialized, values is empty.
First question: why values is empty and when it is filled with the values?
Second question: how can I implement prevOf and nextOf?
Third question: is it possible to add the methods prevOf and nextOf to the value type instead of the enumeration? Writing One.next feels more natural than writing Num.nextOf(One)
try the following codes:
class PrevNextEnum extends Enumeration {
lazy val prevOf = {
val list = values.toList
val map = list.tail.zip(list.map(Some(_))).toMap + (list.head -> None)
map
}
lazy val nextOf = {
val list = values.toList
val map = (list.zip(list.tail.map(Some(_)) :+ None).toMap)
map
}
}
object Nums extends PrevNextEnum {
type Nums = Value
val One, Two, Three = Value
}
object App extends App {
println(Nums.prevOf(Nums.Two))
println(Nums.nextOf(Nums.One))
println(Nums.nextOf(Nums.Three))
println(Nums.prevOf(Nums.One))
}
Building on the answer of user1484819 :
class PrevNextEnum extends Enumeration {
lazy val prevOf = {
val list = values.toList
val map = list.tail.zip(list).toMap
v:Value => map.get(v)
}
lazy val nextOf = {
val list = values.toList
val map = list.zip(list.tail).toMap
v:Value => map.get(v)
}
}
object Nums extends PrevNextEnum {
type Nums = Value
val One, Two, Three = Value
}
This has basically the same structure, but uses the fact that Map can return Options itself when using get instead of apply.
Related
I've come across an issue where using the this keyword in Scala multiple times within the same method will actually fail.
I cannot replicate the problem here because the codebase is too large, but I will do my best to show the problem.
I have a class, Foo, with an override-able method fooMethod. Inside fooMethod, three property values are generated, and the current class instance is updated with these properties separately, using this. However, only the last this call actually sets any of the properties, the two preceding this calls have no effect on the object.
case class Foo(prop1: prop, prop2: prop2, prop3:prop3) extends FooParent {
override def fooMethod(){
val propA = gen()
val propB = gen()
val propC = gen()
this.withPropA(propA)
this.withPropB(propB)
this.withPropC(propC)
}
def withPropA(:propA): Foo = this.copy(prop1 = propA)
def withPropB(:propB): Foo = this.copy(prop2 = propB)
def withPropC(:propC): Foo = this.copy(prop3 = propC)
}
The above code will only apply the final this.withPropC call, and therefore only prop3 is updated.
However, if I do the following
case class Foo(prop1: prop, prop2: prop2, prop3:prop3) extends FooParent {
override def fooMethod(){
val propA = gen()
val propB = gen()
val propC = gen()
// here
val one = this.withPropA(propA)
val two = one.withPropB(propB)
two.withPropC(propC)
}
def withPropA(:propA): Foo = this.copy(prop1 = propA)
def withPropB(:propB): Foo = this.copy(prop2 = propB)
def withPropC(:propC): Foo = this.copy(prop3 = propC)
}
then all of the properties are updated. Why is this?
Case class are immutable, so copy based functions (like those withProp*) never mutate this.
Moreover, in case of sequential property changes, each update must be applied on the result of previous one (chain); Not on (orginal) this, what would create separate instances corresponding to each unrelated update.
If you really intend to use such functions, it should be
withPropA(propA). // chain update
withPropB(propB). // chain
withPropC(propC)
What can be written:
copy(prop1 = propA, prop2 = propB, prop3 = propC)
we can see StateStoreRestoreExec as follows.
case class StateStoreRestoreExec(
keyExpressions: Seq[Attribute],
stateId: Option[OperatorStateId],
child: SparkPlan)
extends UnaryExecNode with StateStoreReader {
override protected def doExecute(): RDD[InternalRow] = {
val numOutputRows = longMetric("numOutputRows")
child.execute().mapPartitionsWithStateStore(
getStateId.checkpointLocation,
operatorId = getStateId.operatorId,
storeVersion = getStateId.batchId,
keyExpressions.toStructType,
child.output.toStructType,
sqlContext.sessionState,
Some(sqlContext.streams.stateStoreCoordinator)) { case (store, iter) =>
val getKey = GenerateUnsafeProjection.generate(keyExpressions, child.output)
iter.flatMap { row =>
val key = getKey(row)
val savedState = store.get(key)
numOutputRows += 1
row +: savedState.toSeq
}
}
Here, I wonder the meaning of row +: savedState.toSeq . I think row is a instance of UnsafeRow and savedState.toSeq is a instance of Seq. So how can we operate them with +:. On the other hand, I think savedState is a instance of UnsafeRow and toSeq is not a member of UnsaveRow, So how does savedState.toSeq work?
row is an instance of an InternalRow, and savedState is an Option[UnsafeRow], which extends InternalRow. What happens here is that the saved state is transformed from Option[UnsafeRow] to a Seq[UnsafeRow] and then the row instance is prepended to that sequence.
When you flatMap over these UnsafeRow objects, you get back an Iterator[UnsafeRow].
I have this custom Scala object (basically a Java POJO):
object CustomObject {
implicit object Mapper extends JavaBeanColumnMapper[CustomObject]
}
class CustomObject extends Serializable {
#BeanProperty
var amount: Option[java.lang.Double] = _
...
}
In my main class, I've loaded an RDD that contains these CustomObjects.
I am trying to filter them and create a new RDD that contains only the objects that have amount > 5000.
val customObjectRDD = sc.objectFile[CustomObject]( "objectFiles" )
val filteredRdd = customObjectRDD.filter( x => x.amount > 5000 )
println( filteredRdd.count() )
However, my editor says
Type Mismatch: Expected: (CustomObject) => Boolean, Actual:
(CustomObject) => Any
What do I have to do to get this to work?
The > operator is not defined on Option[Double], your filter predicate will need to handle the Option:
scala> case class A(amount: Option[Double])
defined class A
scala> val myRDD = sc.parallelize(Seq(A(Some(10000d)), A(None), A(Some(5001d)), A(Some(5000d))))
myRDD: org.apache.spark.rdd.RDD[A] = ParallelCollectionRDD[12] at parallelize at <console>:29
scala> myRDD.filter(_.amount.exists(_ > 5000)).foreach{println}
A(Some(10000.0))
A(Some(5001.0))
This assumes that any object with amount = None should fail the filter predicate. See the docs for a definition of Option.exists.
Say i have enum like this:
object Enm extends Enumeration {
val ONE, TWO, THREE = Value
}
and it's possible for me to get disired result by calling with name on it Enm.withName("ONE"), but if i have Value with argument, say:
object Enm extends Enumeration {
val ONE = Value("1")
val TWO = Value("2")
val THREE = Value("3")
}
calling Enm.withName("ONE") i get back noting since it's now "1", "2" and so on.
Is it possible somehow to get val names but not the ones i put in Value?
Hacky solution
Maintain a Map of word string to Number string mapping
object Enm extends Enumeration {
val ONE = Value("1")
val TWO = Value("2")
val THREE = Value("3")
val map = Map ("ONE" -> "1", "TWO" -> "2", "THREE" -> "3")
}
//Usage
Enm.withName(Enm.map("ONE"))
You cannot override withName method as it is final. So write a custom withName method
object Enm extends Enumeration {
val ONE = Value("1")
val TWO = Value("2")
val THREE = Value("3")
private val map = Map ("ONE" -> "1", "TWO" -> "2", "THREE" -> "3")
def customWithName(str: String) = withName(map(str))
}
//Usage
Enm.customWithName("ONE")
I have a simple Scala class like this:
class FiltersBuilder {
def build(filter: CommandFilter) = {
val result = collection.mutable.Map[String, String]()
if (filter.activity.isDefined) {
result += ("activity" -> """ some specific expression """)
} // I well know that manipulating option like this is not recommanded,
//it's just for the simplicity of the example
if (filter.gender.isDefined) {
result += ("gender" -> """ some specific expression """)
}
result.toMap //in order to return an immutable Map
}
}
using this class so:
case class CommandFilter(activity: Option[String] = None, gender: Option[String] = None)
The result content depends on the nature of the selected filters and their associated and hardcoded expressions (String).
Is there a way to transform this code snippet by removing this "mutability" of the mutable.Map?
Map each filter field to a tuple while you add the result to a Seq, then filter out the Nones with flatten finally convert the Seq of tuples to a Map with toMap.
For adding more fields to filter you just have to add a new line to the Seq
def build(filter: CommandFilter) = {
// map each filter filed to the proper tuple
// as they are options, map will transform just the Some and let the None as None
val result = Seq(
filter.activity.map(value => "activity" -> s""" some specific expression using $value """),
filter.gender.map(value => "gender" -> s""" some specific expression using $value """)
).flatten // flatten will filter out all the Nones
result.toMap // transform list of tuple to a map
}
Hope it helps.
Gaston.
Since there are at most 2 elements in your Map:
val activity = filter.activity.map(_ => Map("activity" -> "xx"))
val gender = filter.gender.map(_ => Map("gender" -> "xx"))
val empty = Map[String, String]()
activity.getOrElse(empty) ++ gender.getOrElse(empty)
I've just managed to achieve it with this solution:
class FiltersBuilder(commandFilter: CommandFilter) {
def build = {
val result = Map[String, String]()
buildGenderFilter(buildActivityFilter(result))
}
private def buildActivityFilter(expressions: Map[String, String]) =
commandFilter.activity.fold(expressions)(activity => result + ("activity" -> """ expression regarding activity """))
private def buildGenderFilter(expressions: Map[String, String]) =
commandFilter.gender.fold(expressions)(gender => result + ("gender" -> """ expression regarding gender """))
}
Any better way?