Race condition caused by mvar - scala

Consider the following simple example:
implicit val timer: Timer[IO] = IO.timer(ExecutionContext.global)
implicit val cs: ContextShift[IO] = IO.contextShift(ExecutionContext.global)
val mvarF = MVar.of[IO, mutable.Map[Int, Int]](mutable.Map.empty)
mvarF.flatMap(mvar =>
mvar.take.bracket(st => {
IO(st.put(1, 1)) >> (IO.sleep(2.seconds) >> IO(st.clear())).start
})(mvar.put)
>>
mvar.take.bracket(st =>
IO(println(s"Size before sleep ${st.size}")) >> IO.sleep(2.seconds) >> IO(println(s"Size after sleep ${st.size}"))
)(mvar.put)
).unsafeRunSync()
It prints:
Size before sleep 1
Size after sleep 0
In this example the job scheduled under fiber modifies the object under mvar which is acquired by another job.
This extremely unsafe. Is there a way to prohibit such usage?

As pointed by Jasper, your main problem (for this specific code example) is that you are releasing the bracket after starting a new fiber in your use by calling IO(st.put(1, 1)) >> (IO.sleep(2.seconds) >> IO(st.clear())).start. So your use is actually an mutable.Map[Int, Int] => IO[Fiber[IO, Unit]].
You just have to remove that start and you will have intended behaviour (your use will be mutable.Map[Int, Int] => IO[Unit] and bracket will not be released unless your use IO completes.). This means that the map will alreay be empty for both print actions.
mvarF.flatMap(mvar =>
mvar.take.bracket(st => {
IO(st.put(1, 1)) >>
IO.sleep(2.seconds) >>
IO(st.clear())
})(mvar.put)
>>
mvar.take.bracket(st =>
IO(println(s"Size before sleep ${st.size}")) >>
IO.sleep(2.seconds) >>
IO(println(s"Size after sleep ${st.size}"))
)(mvar.put)
).unsafeRunSync()
Size before sleep 0
Size after sleep 0
But this is actually just a conincidence for this specific code example (IO's are being chained with flatMap which means we are telling the runtime to sequentially perform these IO's).
MVar provides you control over re-assignment of the variable, but your are not doing any re-assignment at all. Hence, this code is not even using any capbilities of MVar, its just sitting there as an spectator.
So, the usage of MVar in this way will have ZERO impact on thread saftey of your code.
mvarF.flatMap(mvar =>
mvar.take.bracket(st =>
IO(println(s"Size before first sleep - ${st.size}")) >> IO.sleep(2.seconds) >> IO(println(s"Size after first sleep - ${st.size}"))
)(mvar.put)
).unsafeRunAsyncAndForget()
mvarF.flatMap(mvar =>
mvar.take.bracket(st => {
IO(st.put(1, 1)) >> IO.sleep(2.seconds) >> IO(st.clear())
})(mvar.put)
).unsafeRunAsyncAndForget()
mvarF.flatMap(mvar =>
mvar.take.bracket(st =>
IO(println(s"Size before second sleep - ${st.size}")) >> IO.sleep(2.seconds) >> IO(println(s"Size after second sleep - ${st.size}"))
)(mvar.put)
).unsafeRunAsyncAndForget()
Size before first sleep - 0
Size before second sleep - 1
Size after first sleep - 0
Size after second sleep - 0
You can use Semaphore to get a race free scope.
class IOWithSemaphore[A](
private val a: A,
private val semaphore: Semaphore[IO]
)(
implicit
F: Concurrent[IO],
T: Timer[IO]) {
def unitUse(use: A => IO[Unit]): IO[Unit] =
for {
_ <- semaphore.acquire
_ <- use(a)
_ <- semaphore.release
} yield ()
}
val map = mutable.Map.empty[Int, Int]
Semaphore[IO](1).map(semaphore => {
val mapIOWithSemaphore = new IOWithSemaphore[mutable.Map[Int, Int]](map, semaphore)
// using unsafeRunAsync to emulate the parallel usage
mapIOWithSemaphore.unitUse(map =>
IO(println(s"Size before first sleep - ${map.size}")) >> IO.sleep(2.seconds) >> IO(println(s"Size after first sleep - ${map.size}"))
).unsafeRunAsyncAndForget()
mapIOWithSemaphore.unitUse(map =>
IO(println(s"MUTATION BEGIN")) >> IO(map.put(1, 1)) >> IO.sleep(2.seconds) >> IO(map.clear()) >> IO(println(s"MUTATION END"))
).unsafeRunAsyncAndForget()
mapIOWithSemaphore.unitUse(map =>
IO(println(s"Size before second sleep - ${map.size}")) >> IO.sleep(2.seconds) >> IO(println(s"Size after second sleep - ${map.size}"))
).unsafeRunAsyncAndForget()
}).unsafeRunAsyncAndForget()
Await.result(Promise[Unit].future, Duration.Inf)
Size before first sleep - 0
Size after first sleep - 0
MUTATION BEGIN
MUTATION END
Size before second sleep - 0
Size after second sleep - 0

Related

How to send the output of the FOR loop as the input to itself in Scala?

Add elements to the array buffer such that the element to be added is the product of the previous two elements of the input array buffer. The elements in the output array buffer should not exceed 10000.
Input ArrayBuffer:- ArrayBuffer(2, 3)
Expected Output ArrayBuffer:- ArrayBuffer(2, 3, 6, 18, 108, 1944)
Actual Output ArrayBuffer:- ArrayBuffer(2, 3, 6, 18)
def main(args: ArrayBuffer[Int]): Unit = {
for(k <- 1 to args.length if(args(k) < 10000)){
args += args(k)*args(k-1)
}
println(args)
}
Here I took args from main function as input to the for block. Then it calculates and appends the result to args only upto the previous length of the input say (2, 3) that is 2. But the length should increase as the elements keep appending to the arraybuffer. Say after the 1st iteration args will be (2, 3, 6, 18). Now this should be fed as input to the FOR block and it should continue till last value reaches closest min value to 10000.
If I find solution for this, I will update here. Until then, please help me solving this 🤞
Following is an adder version,
#tailrec
def adder(args: List[BigDecimal], size: Int = args.length): List[BigDecimal] = {
args.reverse match {
case first :: second :: rest if size < 10000 => adder(((first + second) :: first :: second :: rest).reverse, size + 1)
case _ => args
}
}
But this won't work for multiplication as even a simple input of List(1,2) would crash with ArithmeticException: Overflow at size 48.
EDIT
I had misunderstood the OP as the total number of elements to be less than 10000.
updated version:
#tailrec
def multiplier(args: List[Int]): List[Int] = {
args.reverse match {
case first :: second :: rest if first * second < 10000 => multiplier(((first * second) :: first :: second :: rest).reverse)
case _ => args
}
}
or
def multiplier(args: List[Int]): List[Int] = {
#tailrec
def generate(args: List[Int]): List[Int] = {
args match {
case first :: second :: rest if first * second < 10000 => generate((first * second) :: first :: second :: rest)
case _ => args
}
}
generate(args.reverse).reverse
}
An implementation in FP style would be
2 :: 3 :: List.unfold((2, 3)) { case (x, y) =>
val p = x * y
Option.when(p < 10000)((p, (y, p)))
}
https://github.com/scala/scala/blob/v2.13.10/src/library/scala/collection/Factory.scala#L114-L124
Thanks for your responses. As I like to solve any question in a simple and easy way, I found this one too in same manner. I've replaced for loop with a while loop and made tiny changes like below
while(args(i)*args(i-1) < 10000){
args += args(i)*args(i-1)
i += 1
}
Now I got the expected Output as I mentioned in my question above
See you all when my next doubt pops up...👋

How can I optimize the timing of this CountingSort algorithm in Scala

I would like to ask you for help to identify which part of my code is not efficient. I am comparing the QuickSort algorithm with the CountingSort algorithm, assuming that the number of elements in an Array[Byte] is less than 16.
However, the CountingSort time is much higher than the QuickSort time, in all the tests I had performed sequentially. Then, I wanted to test this code in Spark to compute the Median Filter, but the results of the distributed execution times are consistent with the sequential execution times. What I mean is that QuickSort is always faster than CountingSort, even for smaller arrays.
Evidently something in my code is hanging the final processing.
This is the code:
def Histogram(Input: Array[Byte]) : Array[Int] = {
val result = Array.ofDim[Int](256)
val range = Input.distinct.map(x => x & 0xFF)
val mx = Input.map(x => x & 0xFF).max
for (h <- range)
result(h) = Input.count(x => (x & 0xFF) == h)
result.slice(0, mx + 1)
}
def CummulativeSum(Input: Array[Int]): Array[Long] = Input.map(x => x.toLong).scanLeft(0.toLong)(_ + _).drop(1)
def CountingSort(Input: Array[Byte]): Array[Byte] = {
val hist = Histogram(Input)
val cum = CummulativeSum(hist)
val Output = Array.fill[Byte](Input.length)(0)
for (i <- Input.indices) {
Output(cum(Input(i) & 0xFF).toInt - 1) = Input(i)
cum(Input(i) & 0xFF) -= 1
}
Output
}
You can build your histogram without traversing the input quite so many times.
def histogram(input :Array[Byte]) :Array[Int] = {
val inputMap :Map[Int,Array[Byte]] = input.groupBy(_ & 0xFF)
.withDefaultValue(Array())
Array.tabulate(inputMap.keys.max+1)(inputMap(_).length)
}
I'm not sure if this is much faster, but it is certainly more concise.
def countingSort(input :Array[Byte]) :Array[Byte] =
histogram(input).zipWithIndex.flatMap{case (v,x) => Seq.fill(v)(x.toByte)}
My tests show it produces the same results but there could be edge conditions that I've missed.

Scala functional code for iteration with a value forwarded to next iteration

How to make this code more functional, where a value from the current iteration is preserved into the next (variable last),
def f(i: Int): Int = ???
var last = -1
for (v <- xs) yield {
val start = f(v)
val end = f(start)
val res = if (last == start - 2) "abc" else "xyz"
last = end
res
}
I think foldLeft should work. You can pass the value that you will need in the next iteration and the result in a tuple
val list = (1 to 10 by 2)
//> list : scala.collection.immutable.Range = Range(1, 3, 5, 7, 9)
val last = -1
list.foldLeft((last, List[String]())) {
case (r, c) => println(s"Current - $c last - ${r._1}")
(c, r._2 :+ if (...) "abc" else "xyz")
// c will be the value that you need to pass to the next iteration
// r._2 will contain the list which you would have got with your for comprehension
}
//> Current - 1 last - -1
//| Current - 3 last - 1
//| Current - 5 last - 3
//| Current - 7 last - 5
//| Current - 9 last - 7
Noticed this approach with scanLeft (inspired in #mohit approach using foldLeft), here on a pre-processed collection of pairs for start and end, namely for instance
val a = Array((1,2), (3,4), (5,6))
Then in what follows, assume s stands for start and e for end, and so
a.scanLeft((-1,-1,-1)) { case(acc, (s,e)) => (s,e,acc._2) }.drop(1)
which produces triplets with start, end and last,
Array((1,2,-1), (3,4,2), (5,6,4))
over which we can apply the if-else expression; altogether,
a.scanLeft((-1,-1,-1)) { case(acc, (s,e)) => (s,e,acc._2) }.
drop(1).
map(t => if (t._3 == t._1 - 2) "abc" else "xyz")

Akka with Frege running slower than Scala counterpart

As an exercise, I took these Scala and Java examples of Akka to port to Frege. While it works fine, it runs slower(11s) than Scala(540ms) counterpart.
module mmhelloworld.akkatutorialfregecore.Pi where
import mmhelloworld.akkatutorialfregecore.Akka
data PiMessage = Calculate |
Work {start :: Int, nrOfElements :: Int} |
Result {value :: Double} |
PiApproximation {pi :: Double, duration :: Duration}
data Worker = private Worker where
calculatePiFor :: Int -> Int -> Double
calculatePiFor !start !nrOfElements = loop start nrOfElements 0.0 f where
loop !curr !n !acc f = if n == 0 then acc
else loop (curr + 1) (n - 1) (f acc curr) f
f !acc !i = acc + (4.0 * fromInt (1 - (i `mod` 2) * 2) / fromInt (2 * i + 1))
onReceive :: Mutable s UntypedActor -> PiMessage -> ST s ()
onReceive actor Work{start=start, nrOfElements=nrOfElements} = do
sender <- actor.sender
self <- actor.getSelf
sender.tellSender (Result $ calculatePiFor start nrOfElements) self
data Master = private Master {
nrOfWorkers :: Int,
nrOfMessages :: Int,
nrOfElements :: Int,
listener :: MutableIO ActorRef,
pi :: Double,
nrOfResults :: Int,
workerRouter :: MutableIO ActorRef,
start :: Long } where
initMaster :: Int -> Int -> Int -> MutableIO ActorRef -> MutableIO UntypedActor -> IO Master
initMaster nrOfWorkers nrOfMessages nrOfElements listener actor = do
props <- Props.forUntypedActor Worker.onReceive
router <- RoundRobinRouter.new nrOfWorkers
context <- actor.getContext
workerRouter <- props.withRouter router >>= (\p -> context.actorOf p "workerRouter")
now <- currentTimeMillis ()
return $ Master nrOfWorkers nrOfMessages nrOfElements listener 0.0 0 workerRouter now
onReceive :: MutableIO UntypedActor -> Master -> PiMessage -> IO Master
onReceive actor master Calculate = do
self <- actor.getSelf
let tellWorker start = master.workerRouter.tellSender (work start) self
work start = Work (start * master.nrOfElements) master.nrOfElements
forM_ [0 .. master.nrOfMessages - 1] tellWorker
return master
onReceive actor master (Result newPi) = do
let (!newNrOfResults, !pi) = (master.nrOfResults + 1, master.pi + newPi)
when (newNrOfResults == master.nrOfMessages) $ do
self <- actor.getSelf
now <- currentTimeMillis ()
duration <- Duration.create (now - master.start) TimeUnit.milliseconds
master.listener.tellSender (PiApproximation pi duration) self
actor.getContext >>= (\context -> context.stop self)
return master.{pi=pi, nrOfResults=newNrOfResults}
data Listener = private Listener where
onReceive :: MutableIO UntypedActor -> PiMessage -> IO ()
onReceive actor (PiApproximation pi duration) = do
println $ "Pi approximation: " ++ show pi
println $ "Calculation time: " ++ duration.toString
actor.getContext >>= ActorContext.system >>= ActorSystem.shutdown
calculate nrOfWorkers nrOfElements nrOfMessages = do
system <- ActorSystem.create "PiSystem"
listener <- Props.forUntypedActor Listener.onReceive >>= flip system.actorOf "listener"
let constructor = Master.initMaster nrOfWorkers nrOfMessages nrOfElements listener
newMaster = StatefulUntypedActor.new constructor Master.onReceive
factory <- UntypedActorFactory.new newMaster
masterActor <- Props.fromUntypedFactory factory >>= flip system.actorOf "master"
masterActor.tell Calculate
getLine >> return () --Not to exit until done
main _ = calculate 4 10000 10000
Am I doing something wrong with Akka or is it something to do with laziness in Frege for being slow? For example, when I initially had fold(strict fold) in place of loop in Worker.calculatePiFor, it took 27s.
Dependencies:
Akka native definitions for Frege: Akka.fr
Java helper to extend Akka classes since we cannot extend a class in
Frege: Actors.java
I am not exactly familiar with Actors, but assuming that the tightest loop is indeed loop you could avoid passing function f as argument.
For one, applications of passed functions cannot take advantage of the strictness of the actual passed function. Rather, code generation must assume conservatively that the passed function takes its arguments lazily and returns a lazy result.
Second, in our case you use f really just once here, so one can inline it. (This is how it is done in the scala code in the article you linked.)
Look at the code generated for the tail recursion in the following sample code that mimics yours:
test b c = loop 100 0 f
where
loop 0 !acc f = acc
loop n !acc f = loop (n-1) (acc + f (acc-1) (acc+1)) f -- tail recursion
f x y = 2*x + 7*y
We get there:
// arg2$f is the accumulator
arg$2 = arg$2f + (int)frege.runtime.Delayed.<java.lang.Integer>forced(
f_3237.apply(PreludeBase.INum_Int._minusÆ’.apply(arg$2f, 1)).apply(
PreludeBase.INum_Int._plusÆ’.apply(arg$2f, 1)
).result()
);
You see here that f is called lazily which causes all the argument expressios to also be computed lazily. Note the number of method calls this requires!
In your case the code should still be something like:
(double)Delayed.<Double>forced(f.apply(acc).apply(curr).result())
This means, two closures are build with the boxed values acc and curr and then the result is computed, i.e. the function f gets called with the unboxed arguments, and the result gets again boxed, just to get unboxed again (forced) for the next loop.
Now compare the following, where we just do not pass f but call it directly:
test b c = loop 100 0
where
loop 0 !acc = acc
loop n !acc = loop (n-1) (acc + f (acc-1) (acc+1))
f x y = 2*x + 7*y
We get:
arg$2 = arg$2f + f(arg$2f - 1, arg$2f + 1);
Much better!
Finally, in the case above we can do without a function call at all:
loop n !acc = loop (n-1) (acc + f) where
f = 2*x + 7*y
x = acc-1
y = acc+1
And this gets:
final int y_3236 = arg$2f + 1;
final int x_3235 = arg$2f - 1;
...
arg$2 = arg$2f + ((2 * x_3235) + (7 * y_3236));
Please try this out and let us know what happens. The main boost in performance should come from not passing f, whereas the inlining will probably be done in the JIT anyway.
The additional cost with fold is probably because you also had to create some list before applying it.

Does isEmpty method in Stream evaluate the whole Stream?

In Scala, does calling isEmtpy method on an instance of Stream class cause the stream to be evaluated completely? My code is like this:
import Stream.cons
private val odds: Stream[Int] = cons(3, odds.map(_ + 2))
private val primes: Stream[Int] = cons(2, odds filter isPrime)
private def isPrime(n: Int): Boolean = n match {
case 1 => false
case 2 => true
case 3 => true
case 5 => true
case 7 => true
case x if n % 3 == 0 => false
case x if n % 5 == 0 => false
case x if n % 7 == 0 => false
case x if (x + 1) % 6 == 0 || (x - 1) % 6 == 0 => true
case x => primeDivisors(x) isEmpty
}
import Math.{sqrt, ceil}
private def primeDivisors(n: Int) =
primes takeWhile { _ <= ceil(sqrt(n))} filter {n % _ == 0 }
So, does the call to isEmpty on the line case x => primeDivisors(x) isEmpty cause all the prime divisors to be evaluated or only the first one?
Only if the stream is actually empty :)
Otherwise, it will just see if the stream has a head and tail (matches Stream.cons) and return false.
Looking at the source:
https://lampsvn.epfl.ch/trac/scala/browser/scala/trunk/src///library/scala/collection/immutable/Stream.scala#L550
would suggest that once you have a Stream.Cons instance it cannot be empty, so always returns false. It therefore seems likely that the Stream.cons factory method is the one that evaluates the first element, which you can show by running just:
Stream.cons(print("Hello"), Stream(print(" World!")))
in the interpreter and seeing this also prints "Hello".