Here is a code segment of scala. I set timeout as 100 mills. Out of 10000 loops, 106 of them run more than 100 mills without throwing exceptions. The largest one is even 135 mills. Any reason why this is happening?
for (j <- 0 to 10000) {
total += 1
val executor = Executors.newSingleThreadExecutor
val result = executor.submit[Int](new Callable[Int] {
def call = try {
Thread.sleep(95)
for (i <- 0 to 1000000) {}
4
} catch {
case e: Exception => exception1 += 1
5
}
})
try {
val t1 = Calendar.getInstance.getTimeInMillis
result.get(100, TimeUnit.MILLISECONDS)
val t2 = Calendar.getInstance.getTimeInMillis
println("timediff = " + (t2 - t1).toString)
} catch {
case e: Exception => exception2 += 1
}
}
Firstly, if you're running on Windows you should be aware that the timer resolution is around 15.6 milliseconds.
Secondly, your empty loop of 1M iterations is quite likely to be removed by a compiler, and more importantly, can't be interrupted by any timeout.
The way a thread sleep works is that a thread asks the o/s to interrupt it after the given time. That's how the timeout works in the result.get call. Now you're relying on the OS thread that does this to be running at the exact time when your timeout has expired, which of course it may not be. Then there is the fact you have 10000 threads for it to interrupt which it can't do all at the exact same time.
Related
I am new to scala check and I want to test the following piece of my application. I want to generate 30 and 20 random events and check if my application code correctly computes a result
// generate 30 random events
val eventGenerator: Gen[Event] = for {
d <- Gen.oneOf[String](Seq("es1", "es2", "es3"))
t <- Gen.choose[Long](minEvent.getTime, maxEvent.getTime)
s <- Gen.oneOf[String](Seq("s1", "s2", "s3", "s4", "s5", "s6", "s7"))} yield Event(d, t, s)
val eventsGenerator: Gen[List[VpSearchLog]] = Gen.containerOfN[List, VpSearchLog](30, eventGenerator)
// generate 20 random instances
val instanceGenerator: Gen[Instance] = for {
d <- Gen.oneOf[String](Seq("es1", "es2", "es3"))
t <- Gen.choose[Long](minInstance.getTime, maxInstance.getTime)} yield Instance(d, new Timestamp(t))
val instancesGenerator: Gen[List[Instance]] = Gen.containerOfN[List, Instance](20, instanceGenerator)
val p: Prop = forAll(instancesGenerator, eventsGenerator) { (i, e) =>
println(i.size)
println(e.size)
println()
val instancesWithFeature = computeExpected(instance)
isEqual(transform(instance), instanceWithFeature)
}
For some reason I see this in the stdout
20
15
20
7
20
3
20
1
20
0
starting to compute expected:
Basically it looks like the forAll generates a couple of inputs with a certain size and then skips them. For some reaon, it starts to compute things when one of the input has size 0 and then it starts the proper check. My questions are:
why if I use containerofN or listOfN I don't get exactly input of that specific size? How can I then generate input like this?
is it normal that forAll starts to explore the space of the possible input and skips some of them? Am I missing something here? This behaviour is quite counter intuitive for me
You may need to use forAllNoShrink to avoid the known defect in ScalaCheck where shrinking fails to respect generators
val thirtyInts: Gen[List[Int]] =
Gen.listOfN[Int](30, Gen.const(99))
val twentyLongs: Gen[List[Long]] =
Gen.listOfN[Long](20, Gen.const(44L))
property("listOfN") = {
Prop.forAllNoShrink(thirtyInts, twentyLongs) { (ii, ll) =>
ii.size == 30 && ll.size == 20
}
}
I am running the following snippet in Spark, trying to see a speed-up?
val howManyPrimes = sc.paralellize(1 to 1000000).filter(isPrime).count()
Where isPrime is implemented used trial division.
I run spark-shell:
spark-shell --master=local[8]
I wait for it to startup, and that snippet runs in ~50 seconds.
Then I quit it and open a spark-shell again but now with 1 core:
spark-shell --master=local[1]
And I again the snippet runs in ~50 seconds. Why?
isPrime is implemented as follows:
def isPrime(x: Int):Boolean = {
var d = 2
while(d < x) {
if (x%d == 0)
return false
else d += 1
}
return true
}
I'm running a Spark job to aggregate data. I have a custom data structure called a Profile, which basically contains a mutable.HashMap[Zone, Double]. I want to merge all profiles that share a given key (a UUID), with the following code:
def merge = (up1: Profile, up2: Profile) => { up1.addWeights(up2); up1}
val aggregated = dailyProfiles
.aggregateByKey(new Profile(), 3200)(merge, merge).cache()
Curiously, Spark fails with the following error:
org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 116318 tasks (1024.0 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
The obvious solution is to increment "spark.driver.maxResultSize", but two things puzzle me.
Too much of a coincidence that I get 1024.0 greater than 1024.0
All the documentation and help I found googling this particular error and configuration parameter indicates that it affect functions that take a value back to the driver. (say take() or collect()), but I'm not taking ANYTHING to the driver, just reading from HDFS, aggregating, saving back to HDFS.
Does anyone know why I'm getting this error?
Yes, It's failing because The values we see in exception message are
rounded off by one precision and comparison happening in bytes.
That serialized output must be more than 1024.0 MB and less than 1024.1 MB.
Check added Apache Spark code snippet, It's very interesting and very rare to get this error. :)
Here totalResultSize > maxResultSize both are Long types and in holds the value in bytes. But msg holds rounded value from Utils.bytesToString().
//TaskSetManager.scala
def canFetchMoreResults(size: Long): Boolean = sched.synchronized {
totalResultSize += size
calculatedTasks += 1
if (maxResultSize > 0 && totalResultSize > maxResultSize) {
val msg = s"Total size of serialized results of ${calculatedTasks} tasks " +
s"(${Utils.bytesToString(totalResultSize)}) is bigger than spark.driver.maxResultSize " +
s"(${Utils.bytesToString(maxResultSize)})"
logError(msg)
abort(msg)
false
} else {
true
}
}
Apache Spark 1.3 - source
//Utils.scala
def bytesToString(size: Long): String = {
val TB = 1L << 40
val GB = 1L << 30
val MB = 1L << 20
val KB = 1L << 10
val (value, unit) = {
if (size >= 2*TB) {
(size.asInstanceOf[Double] / TB, "TB")
} else if (size >= 2*GB) {
(size.asInstanceOf[Double] / GB, "GB")
} else if (size >= 2*MB) {
(size.asInstanceOf[Double] / MB, "MB")
} else if (size >= 2*KB) {
(size.asInstanceOf[Double] / KB, "KB")
} else {
(size.asInstanceOf[Double], "B")
}
}
"%.1f %s".formatLocal(Locale.US, value, unit)
}
Apache Spark 1.3 - source
I'm trying to recursively define an observable that either emits items from a subject or, if a certain amount of time passes, a default value, in this case I'm using the timer's default value of zero. I'm using RxScala and have begun with the following code:
val s = PublishSubject[Int]()
def o: Observable[Unit] = {
val timeout = Observable.timer(1 second)
Observable.amb(s, timeout)
.first
.concatMap((v) => {
println(v)
o
})
}
ComputationScheduler().createWorker.schedule {
var value = 0
def loop(): Unit = {
Thread.sleep(5000)
s.onNext(value + 1)
value += 1
loop()
}
loop()
}
o.toBlocking.last
This seems like it should work, but the output is confusing. Every other sequence of zeros contains two instead of the expected four. Two zeros are emitted and the remaining three seconds elapses, but without output.
0
0
0
0
1
0
0
2
0
0
0
0
3
0
0
4
0
0
0
0
5
0
0
6
0
0
0
0
7
0
0
8
This one is puzzling indeed! So here's a theory:
In truth, your code is producing 4 ticks every 5 seconds rather than 5.
There is a race condition on the 4th, one, won first by the timeout, then the worker, then the timeout, etc.
So, instead of the sequence being 00001 002 00003... look at it as 0000 1002 0000...
So you might have 2 separate problems here and I can't do much without fiddling with it, but things you can try:
Add also a serial number to o(), so you can see which timouts are not winning the race.
Change the values from 1 and 5 seconds to some that are not multiple of each other, like 1.5 and 5. This might help you take one problem out and focus on the other.
Put an external, unrelated worker to print "----" every second. Start it after 0.3 seconds or so. Might give you a better idea of where the divide is.
Refactoring your code to the following generates the expected result (on my machine):
object Test {
def main(args: Array[String]) {
val s = PublishSubject[Int]()
val timeout = Observable.timer(1 second)
def o: Observable[Unit] = {
Observable.amb(s, timeout).first
.concatMap((v) => {
println(v)
o
})
}
var value = 0
NewThreadScheduler().createWorker.scheduleRec {
Thread.sleep(5000)
value += 1
s.onNext(value)
}
o.toBlocking.last
}
}
Notice the switch to the NewThreadScheduler and the use of the scheduleRec method as opposed to manual recursive scheduling.
I have a function to compute the time spent in a block:
import collection.mutable.{Map => MMap}
var timeTotalMap = MMap[String, Long]()
var numMap = MMap[String, Float]()
var averageMsMap = MMap[String, Float]()
def time[T](key: String)(block: =>T): T = {
val start = System.nanoTime()
val res = block
val total = System.nanoTime - start
timeTotalMap(key) = timeTotalMap.getOrElse(key, 0L) + total
numMap(key) = numMap.getOrElse(key, 0f) + 1
averageMsMap(key) = timeTotalMap(key)/1000000000f/numMap(key)
res
}
I am timing a function and the place where it is called in one place.
time("outerpos") { intersectPos(a, b) }
and the function itself starts with:
def intersectPos(p1: SPosition, p2: SPosition)(implicit unify: Option[Identifier] =None): SPosition
= time("innerpos") {(p1, p2) match {
...
}
When I display the nano times for each key (timeTotalMap), I get (added spaces for lisibility)
outerpos-> 37 870 034 714
innerpos-> 53 647 956
It means that the total execution time of innerpos is 1000 less than the one of outerpos.
What ! there is a factor 1000 between the two ? And its says that the outer call takes 1000x more time than all the inner functions? am I missing something or is there a memory leaking ghost somewhere ?
Update
When I compare the number of execution for each block (numMap), I find the following:
outerpos -> 362878
innerpos -> 21764
This is paradoxal. Even if intersectPos is called somewhere else, shouldn't the number of times it is called be as great as the number of times outerpos is called?
EDIT
If I move the line numMap(key) = numMap.getOrElse(key, 0f) + 1 to the top of the time functinon, then these numbers become approximately equal.
nanoTime on JVM is considered safe but not very accurate. Here are some good reads:
Is System.nanoTime() completely useless?
System.currentTimeMillis vs System.nanoTime
Basically your test will suffer from timer inaccuracy. One way to work around that would be to call time("outerpos") for a quite a long time (note JIT compiler optimization might kick in at some point) and measure start and end times only once. Take an average and repeat with time("innerpos"). That's all I could think of. Still not the best test ever ;)