I am new to pyspark. I noticed that when I run my code in jupyter it shows something like this: (0 + n) / n
when n equals 1, code is running much faster. So my question is what can I do to make it small?
for example, now I am running a code and n equals 8 and it takes forever to run the code.
for example, now I am running a code and n equals 8 and it takes forever to run the code.
Related
In a certain manufacturing system, there are 2 machines M 1 and M 2.
M 1 is a fast and high precision machine whereas M 2 is a slow and low
precision machine. M 2 is employed only when M 1 is down, and it is
assumed that M 2 does not fail. Assume that the processing time of
parts on M 1, the processing time of parts on M 2, the time to failure
of M 1, and the repair time of M 1 are independent geometric random
variables with parameters p 1, p 2, f, and r, respectively. Identify a
suitable state space for the DTMC model of the above system and
compute the TPM. Investigate the steady-state behavior of the DTMC.
How to model this into. DTMC? What is the state space? I have tried to use state-space like this:
0: M1 is working, M1 does not fail
1: M1 failed, M2 is working, M1 is repairing, M1 does not finish repairing
But there are still some problems, like what will happen after M1 finishes 1 part? Immediately process the next one or will change decide whether to fail or not? What will happen if M1 fails during process one part? What is the probability transfer matrix?
Thank you very much for your help!!!!
I want to print integer numbers from 1 to a random integer number (e.g. to a random integer number < 10 like in the following case). I can use e.g. one from the following ways:
1)
val randomNumber=r.nextInt(10)
for (i <- 0 to randomNumber) {println(i)}
2)
for (i <- 0 to r.nextInt(10)) {println(i)}
My question is the following. Is there a difference between 1) and 2) in the sense of computation? It is clear that a random number r.nextInt(10) is computed only once and after that it is assigned to
the variable randomNumber in the first case but what about the second case? Is the part r.nextInt(10) computed only once at the beginnig of the loop or it is computed for each iteration of the loop? If yes then the first variant is better from computation point of view? I know that this proposed example is an easy example but there can be much more complex for loops where the optimization can be very helpful. If the expression r.nextInt(10) in the second case is computed only once what about expressions which are e.g. functions of variable i, something like for (i <- 0 to fce(i)) {println(i)}? I guess that Fce(i) should be evaluated in every iteration of loop.
Thanks for help.
Andrew
No, there are no differences between 1) and 2).
The bound for the for loop will be evaluated once, and then the for loop will repeat this now fixed number of times.
Things would have been very different comparing those two pieces of code :
val randomNumber=r.nextInt(10)
for (i <- 0 to 10) {println(randomNumber)}
and
for (i <- 0 to 10) {println(r.nextInt(10))}
The first for loop will print 10 times the same value, the second will print 10 random values.
But you could still rewrite the second as
def randomNumber=r.nextInt(10)
for (i <- 0 to 10) {println(randomNumber)}
The change from a val to a function will cause a reevaluation every time the loop is executed.
Now, about
for (i <- 0 to function(i)) {doSomething()}
This doesn't compile. The variable i is only available in the scope between the {} of the for loop, and hence not for computing the loop bounds.
I hope this clarifies things a bit !
The two versions do the same thing, and to see why you need to realise that for in Scala works very differently from how it works in other languages like C/C++.
In Scala, for is just a bit of syntax that makes it easier to chain map/foreach, flatMap, and withFilter methods together. It makes functional code appear more like imperative code, but it is still executed in a functional way.
So your second chunk of code
for (i <- 0 to r.nextInt(10)) {println(i)}
is changed by the compiler into something like
(0 to r.nextInt(10)).foreach{ i => println(i) }
Looking at it this way, you can see that the range is computed first (using a single call to nextInt) and then foreach is called on that range. Therefore nextInt is called the same number of times as in your first chunk of code.
I'd like to use k to calculate the times of the for loop executed. It would be billions of times and I tried long64, then after some time k became negative. Is there any other way to do this?
Sorry I think I made a wrong description. My code is a 3-layer for nest block and each of them is calculating 2*256^3 numbers, once the value is equal to 0, I'd like to make k+=k. In the end I set print, 'k=', k and when idl was running, I found k ran from positive to negative. I used a cluster to do the computation so it didn't take quite a long time.
My guess is that you are not really using a long64 for k. The type for the loop variable in a for loop comes from the start value. For example, in this case:
k = 0LL
for k = 0, n - 1 do ...
k is a int (16-bit) because 0 is a int. You probably want something like:
for k = 0LL, n - 1LL do begin ...
I'm trying to simulate rolling 5 dice for a Yahtzee game I'm writing in MATLAB, but I'm running into the issue that my code doesn't seem to generate any yahtzees after running 1000+ iterations. Am I using a function that will guarantee for a Yahtzee(5 of the same number) to be rolled?
while rounds<=13
fprintf('Rolling the dice...\n');
roll=randi(6,1,5);
roll=sort(roll);
fprintf('You rolled:');
disp(roll);
rollCount=rollCount+1;
for x=rule
if roll==rule{1};
fprintf('Condition Met');
break;
end
end
rounds=rounds+1;
end
This basically just iterates through 13 rounds of rolling and checks the roll against "rule{1}" which is an array that contains [1 1 1 1 1]. There doesn't seem to be a problem getting 3, sometimes 4 1s, but I can't get 5. Any suggestions?
As Ryan Cavanaugh pointed out, there weren't enough samples being done to catch the wanted result. I upped the simulation to run 10000 times and it works as intended.
I recently found out about Parallel Collection in Scala 2.9 and was excited to see that the degree of parallelism can be set using collection.parallel.ForkJoinTasks.defaultForkJoinPool.setParallelism.
However when I tried an experiment of adding two vectors of size one million each , I find
Using parallel collection with parallelism set to 64 is as fast as sequential (Shown in results).
Increasing setParallelism seems to increase performance in a non-linear way. I would have atleast expected monotonic behaviour (That is performance should not degrade if I increase parallelism)
Can some one explain why this is happening
object examplePar extends App{
val Rnd = new Random()
val numSims = 1
val x = for(j <- 1 to 1000000) yield Rnd.nextDouble()
val y = for(j <- 1 to 1000000) yield Rnd.nextDouble()
val parInt = List(1,2,4,8,16,32,64,128,256)
var avg:Double = 0.0
var currTime:Long = 0
for(j <- parInt){
collection.parallel.ForkJoinTasks.defaultForkJoinPool.setParallelism(j)
avg = 0.0
for (k <- 1 to numSims){
currTime = System.currentTimeMillis()
(x zip y).par.map(x => x._1 + x._2)
avg += (System.currentTimeMillis() - currTime)
}
println("Average Time to execute with Parallelism set to " + j.toString + " = "+ (avg/numSims).toString + "ms")
}
currTime = System.currentTimeMillis()
(x zip y).map(x => x._1 + x._2)
println("Time to execute using Sequential = " + (System.currentTimeMillis() - currTime).toString + "ms")
}
The results on running the example using Scala 2.9.1 and a four core processor is
Average Time to execute with Parallelism set to 1 = 1047.0ms
Average Time to execute with Parallelism set to 2 = 594.0ms
Average Time to execute with Parallelism set to 4 = 672.0ms
Average Time to execute with Parallelism set to 8 = 343.0ms
Average Time to execute with Parallelism set to 16 = 375.0ms
Average Time to execute with Parallelism set to 32 = 391.0ms
Average Time to execute with Parallelism set to 64 = 406.0ms
Average Time to execute with Parallelism set to 128 = 813.0ms
Average Time to execute with Parallelism set to 256 = 469.0ms
Time to execute using Sequential = 406ms
Though these results are for one run, they are consistent when averaged over more runs
Parallelism does not come free. It requires extra cycles to split the problem into smaller chunks, organize everything, and synchronize the result.
You can picture this as calling all your friends to help you move, waiting for them to get there, helping you load the truck, then taking them out to lunch, and finally, getting on with your task.
In your test case you are adding two doubles, which is a trivial exercise and takes so little time that overhead from parallelization is greater than simply doing the task in a one thread.
Again, the analogy would be to call all your friends to help you move 3 suitcases. It would take you half a day to get rid of them, while you could finish by yourself in minutes.
To get any benefit from parallelization your task has to be complicated enough to warrant the extra overhead. Try doing some expensive calculations, for example a formula involving a mix of 5-10 trigonometric and logarithmic functions.
I would suggest studying and using the scala.testing.Benchmark trait to benchmark snippets of code. You have to take JIT, GC and other things into account when benchmarking on the JVM - see this paper. In short, you have to do each of the runs in the separate JVM after doing a couple of warm-up runs.
Also, note that the (x zip y) part does not occur in parallel, because x and y are not yet parallel - the zip is done sequentially. Next, I would suggest turning x and y into arrays (toArray) and then calling par - this will ensure that the benchmark uses parallel arrays, rather than parallel vectors (which are slower for transformer methods such as zip and map).