Let's say that there is a boolean variable, initially assigned the value false. Is there a way to say, let this variable be true for the next 5 minutes and then let it be false again?
My current idea is that I could store the variable along with a timestamp, then as soon as the variable is turned to true, the timestamp is set to the current time, then check the variable's value through a method that will return true if the current time and the initial timestamp form a duration of less than 5 minutes and then false if the duration is greater than 5 minutes. I'm thinking about an implicit conversion from Boolean to a RichBoolean that would handle this mechanism through an elegant method name or something.
Is there a more elegant or Scala native way of doing this?
The timestamp idea is pretty easy to implement (here the argument is in seconds):
class TransientlyTrue(duration: Double) {
private[this] val createdAt = System.nanoTime
def value = (System.nanoTime-createdAt)*1e-9 <= duration
}
implicit def TransientToBoolean(t: TransientlyTrue) = t.value
With the implicit conversion you can transparently drop this in wherever you need a Boolean. Or you could leave that off and just call .value.
You may be able to accomplish what you want more elegantly with futures. You won't have a proper variable that's changing its value, but it's not clear that that's an absolute requirement in your question, and you will have a method that starts returning a different value after a certain amount of time. For example, with Twitter's implementation:
import com.twitter.util._
import com.twitter.conversions.time._
implicit val timer = new JavaTimer
val t = Future.sleep(10.seconds)
Now t.isDone will be false for ten seconds and then true. The nice thing about this approach is that you get lots of other combinators on the future that may allow you to solve your problem more directly—see the Scala documentation for more information.
Would you be okay with a wrapper object for this? Like TimedBoolean? If so, then you could do something pretty straight forward with either a Timer, or just on your getter and setter do a timestamp check..
To directly answer your question, though, no, there's no Scala native way of doing this. But I'd think that a wrapper method is the "more elegant" way you're looking for.
Related
I'm creating one report for all simulations in SCALA, in part of them I don't have all attributes and I need to replace that because I'm getting no attribute defined error.
I was trying below code and many many more:
val dbc_communityID = if ("${communityID.exists()}" == true) "${communityID}" else "N/A"
I always have "N/A". I also checked
== 1
== "true"
I spend definitely to much time of that,
Please help
Quoting Gatling official documentation:
This Expression Language only works on String values being passed to Gatling DSL
methods. Such Strings are parsed only once, when the Gatling
simulation is being instantiated.
For example queryParam("latitude", session => "${latitude}") wouldn’t
work because the parameter is not a String, but a function that
returns a String.
Also, queryParam("latitude", "${latitude}".toInt) wouldn’t because the
toInt would happen before passing the parameter to the queryParam
method.
The solution here would be to pass a function:
session => session("latitude").validate[Int].
In short: you can't mix Gatling EL and functions/custom code. If you want to implement some custom logic that deals with Session data, you have to use the Session API, eg:
session => session("communityID").asOption[String].getOrElse("N/A")
if ("${communityID.exists()}" == true)
You are comparing a fixed String with a Boolean so the result will always be false. There is probably a compiler warning about this.
It is possible that you want this, but hard to say without know more about the rest of the code:
if (s"${communityID.exists()}" == "true")
But in that case it is simpler to write
if (communityID.exists())
I have been using the mapWithState API in Spark Streaming, but 2 things are not clear about the StateSpec.function:
Let's say my function is:
def trackStateForKey(batchTime: Time,
key: Long,
newValue: Option[JobData],
currentState: State[JobData]): Option[(Long, JobData)]
Why is the new value an Option[T] type? As far as I've seen, it was always defined for me, and since the method is supposed to be called with a new state, I don't really see the point why it could be optional.
What does the return value mean? I tried to find some pointers in the documentations and source code, but none of them describe what it is used for. Since I'm modifying the state of a key using state.remove() and state.update(), why would I have to do the same with return values?
In my current implementation I return None if I remove the key, and Some(newState) if I update it, but I'm not sure if that is correct.
Why is the new value an Option[T] type? As far as I've seen, it was
always defined for me, and since the method is supposed to be called
with a new state, I don't really see the point why it could be
optional.
It is an Option[T] for the reason that if you set a timeout using StateSpec.timeout, e.g:
StateSpec.function(spec _).timeout(Milliseconds(5000))
then the value passed in once the function times out will be None and the isTimingOut method on State[T] will yield true. This makes sense, because a timeout of the state doesn't mean that a new value has arrived for the specified key, and generally safer to use than passing null for T (which wouldn't work for primitives anyway) as you expect the user to safely operate on an Option[T].
You can see that in the Sparks implementation:
// Get the timed out state records, call the mapping function on each and collect the
// data returned
if (removeTimedoutData && timeoutThresholdTime.isDefined) {
newStateMap.getByTime(timeoutThresholdTime.get).foreach { case (key, state, _) =>
wrappedState.wrapTimingOutState(state)
val returned = mappingFunction(batchTime, key, None, wrappedState) // <-- This.
mappedData ++= returned
newStateMap.remove(key)
}
}
What does the return value mean? I tried to find some pointers in the
documentations and source code, but none of them describe what it is
used for. Since I'm modifying the state of a key using state.remove()
and state.update(), why would I have to do the same with return
values?
The return value is a way to pass intermediate state along the spark graph. For example, assume that I want to update my state but also perform some operation in my pipeline with the intermediate data, e.g:
dStream
.mapWithState(stateSpec)
.map(optionIntermediateResult.map(_ * 2))
.foreachRDD( /* other stuff */)
That return value is exactly what allows me to continue operating on said data. If you don't care for the intermediate result and only want the complete state, then outputting None is perfectly fine.
Edit:
I've written a blog post (following this question) which attempts to give an in-depth explanation to the API.
This is probably a very silly question, but I have a case class which takes as a parameter Option[Timestamp]. The reason this is necessary is because sometimes the timestamp isn't included. However, for testing purposes I'm making an object where I pass in
Timestamp.valueOf("2016-01-27 22:27:32.596150")
But, it seems I can't do this as this is an actual Timestamp, and it's expecting an Option.
How do I convert to Option[Timestamp] from Timestamp. Further, why does this cause a problem to begin with? Isn't the whole benefit of Option that it could be there or not?
Thanks in advance,
Option indicates the possibility of a missing value, but you still need to construct an Option[Timestamp] value. There are two subtypes for Option - None when there is no value, and Some[T] which contains a value of type T.
You can create one directly using Some:
Some(Timestamp.valueOf("2016-01-27 22:27:32.596150"))
or Option.apply:
Option(Timestamp.valueOf("2016-01-27 22:27:32.596150"))
I' was curious if the hashCode() function returns always the same value for a lambda or something in Scala?
My tests have shown to me some static value that does not change even over builds. Is this intended behavior or may it change in future?
If this was some static behavior it would help me a lot building my library.
EDIT:
Let's take this source code:
object Main {
def main(args: Array[String]): Unit = {
val x = (s: String) => 1
val y = (s: String) => 2
println(x.hashCode())
println(y.hashCode())
}
}
It's output on console is for me always 1792393294 and 226170135.
What I'm currently doing is implementing a parser combinator library which I implemented in several languages. And I need to know when wrapper classes are the same (e.g. the underlying functions are the same) so I can implement something like a call stack, which I need to parse as far as possible on failure but prevent endless recursion on error.
Thanks in advance!
The default implementation for hashCode (at least in the Oracle JVM) is in terms of the (initial) memory address of that particular object. So if your program has constructed exactly the same sized objects in exactly the same order before constructing that object, it will in practice return the same value every time.
But this is not at all reliable; most programs do not do exactly the same things every time they run. As soon as you're doing something like responding to user input, that perfect reproducibility will disappear - e.g. maybe you sometimes add enough entries to a HashMap to trigger an enlargement of the table, and sometimes not. And if you construct the same value later in the program, it will of course have a different address; try doing
val z = (s: String) => 1
and observe that it will have a different hashCode from x. Not to mention that the numbers may well be different across different JVMs, different versions of the same JVM, or even when the same JVM is launched with a different -Xms setting.
Computers are often a lot more deterministic in practice than in theory. But this is not the kind of thing that's specified to happen, and certainly not something to rely on in your programs.
I'm integrating with ZeroMQ and Akka to send case classes from different instances. Problem is that when I try to compile this code:
def persistRelay(relayEvent:String, relayData:Any) = {
val relayData = ser.serialize(relayData).fold(throw _, identity)
relayPubSocket ! ZMQMessage(Seq(Frame(relayEvent), Frame(relayData)))
}
The Scala compilers throws back recursive value relayData needs type.
The case classes going in are all different and look like Team(users:List[Long], teamId:Long) and so on.
Is there a method to allow any type in the serializer or a workaround? I'd prefer to avoid writing a serializer for every single function creating the data unless absolutely necessary.
Thanks!
This isn't really a typing issue. The problem is:
val relayData = ser.serialize(relayData).fold(throw _, identity)
You're declaring a val relayData in the same line that you're making a reference to the method parameter relayData. The Scala compiler doesn't understand that you have/want two variables with the same name, and, instead, interprets it as a recursive definition of val relayData. Changing the name of one of those variables should fix the error.
Regardless, since you didn't quite follow what the Scala compiler was asking for, I think that it would also be good to fill you in on what it is that the compiler even wanted from you (even though it's advice that, if followed, probably would have just led to you getting yet another error that wouldn't seem to make a lot of sense, given the circumstances).
It said "recursive value relayData needs type". The meaning of this is that it wanted you to simply specify the type of relayData by having
val relayData = ...
become something like
val relayData: Serializable = ...
(or, in place of Serializable, use whatever type it was that you wanted relayData to have)
It needs this information in order to create a recursive definition. For instance, take the simple case of
val x = x + 1
This code is... bizarre, to say the least, but what I'm doing is defining x in a (shallowly) recursive way. But there's a problem: how can the compiler know what type to use for the inner x? It can't really determine the type through type inference, because type inference involves leveraging the type information of other definitions, and this definition requires x's type information. Now, we might be able to infer that I'm probably talking about an Int, but, theoretically, x could be so many things! In fact, here's the ambiguity in action:
val x: Int = x + 1 // Default value for an Int is '0'
x: Int = 1
val y: String = y + 1 // Default value for a String is 'null'
y: String = null1
All that really changed was the type annotation, but the results are drastically different–and this is only a very simple case! So, yeah, to summarize all this... in most cases, when it's complaining about recursive values needing types, you should just have some empathy on the poor compiler and give it the type information that it so direly craves. It would do the same for you, DeLongey! It would do the same for you!