The throttlelimit parameter of foreach -parallel can control how many processes are used when executing the script. But I can't have more than 5 processes even if I set throttlelimit greater than 5.
The script is executed in multiple powershell processes. So I check PID inside the script. And then group the PIDs so that I can know how many processes are used to execute the script.
function GetPID() {
$PID
}
workflow TestWorkflow {
param($throttlelimit)
foreach -parallel -throttlelimit $throttlelimit ($i in 1..100) {
GetPID
}
}
foreach ($i in 1..8) {
$pids = TestWorkflow -throttlelimit $i
$measure = $pids | group | Measure-Object
$measure.Count
}
The output is
1
2
3
4
5
5
5
5
For $i less or equal to 5, I have $i processes. But for $i greater than 5, I have only 5 processes. Is there any way to increase the number of processes when executing the script?
Edit: In response to #SomeShinyObject's answer, I added another test case. It's a modification of the example given by #SomeShinyObject. I added a function S, which does nothing but sleeping for 10 seconds.
function S($n) {
$s = Get-Date
$s.ToString("HH:mm:ss.ffff") + " start sleep " + $n
sleep 10
$e = Get-Date
$e.ToString("HH:mm:ss.ffff") + " end sleep " + $n + " diff " + (($e-$s).TotalMilliseconds)
}
Workflow Throttle-Me {
[cmdletbinding()]
param (
[int]$ThrottleLimit = 10,
[int]$CollectionLimit = 10
)
foreach -parallel -throttlelimit $ThrottleLimit ($n in 1..$CollectionLimit){
$s = Get-Date
$s.ToString("HH:mm:ss.ffff") + " start " + $n
S $n
$e = Get-Date
$e.ToString("HH:mm:ss.ffff") + " end " + $n + " diff " + (($e-$s).TotalMilliseconds)
}
}
Throttle-Me -ThrottleLimit 10 -CollectionLimit 20
And here's the output. I grouped the output by time (number of seconds), and did a little bit reorder within each group to make it clearer. It's pretty obvious that function S are called 5 by 5, although I set throttlelimit to 10 (first we have start sleep 6..10, 10 seconds later, we have start sleep 1..5, and 10 seconds later start sleep 11..15, and 10 seconds later start sleep 16..20).
03:40:29.7147 start 10
03:40:29.7304 start 9
03:40:29.7304 start 8
03:40:29.7459 start 7
03:40:29.7459 start 6
03:40:29.7616 start 5
03:40:29.7772 start 4
03:40:29.7772 start 3
03:40:29.7928 start 2
03:40:29.7928 start 1
03:40:35.3067 start sleep 7
03:40:35.3067 start sleep 8
03:40:35.3067 start sleep 9
03:40:35.3692 start sleep 10
03:40:35.4629 start sleep 6
03:40:45.3292 end sleep 7 diff 10022.5353
03:40:45.3292 end sleep 8 diff 10022.5353
03:40:45.3292 end sleep 9 diff 10022.5353
03:40:45.3761 end sleep 10 diff 10006.8765
03:40:45.4855 end sleep 6 diff 10022.5243
03:40:45.3605 end 9 diff 15630.1005
03:40:45.3917 end 7 diff 15645.7465
03:40:45.3917 end 8 diff 15661.3313
03:40:45.4229 end 10 diff 15708.2274
03:40:45.5323 end 6 diff 15786.3969
03:40:45.4386 start sleep 5
03:40:45.4542 start sleep 4
03:40:45.4542 start sleep 3
03:40:45.4698 start sleep 2
03:40:45.5636 start sleep 1
03:40:45.4698 start 11
03:40:45.4855 start 12
03:40:45.5011 start 13
03:40:45.5167 start 14
03:40:45.6105 start 15
03:40:55.4596 end sleep 3 diff 10005.4374
03:40:55.4596 end sleep 4 diff 10005.4374
03:40:55.4596 end sleep 5 diff 10021.0426
03:40:55.4752 end sleep 2 diff 10005.3992
03:40:55.5690 end sleep 1 diff 10005.3784
03:40:55.4752 end 3 diff 25698.0221
03:40:55.4909 end 5 diff 25729.302
03:40:55.5065 end 4 diff 25729.2523
03:40:55.5221 end 2 diff 25729.2559
03:40:55.6159 end 1 diff 25823.032
03:40:55.5534 start sleep 11
03:40:55.5534 start sleep 12
03:40:55.5690 start sleep 13
03:40:55.5846 start sleep 14
03:40:55.6784 start sleep 15
03:40:55.6002 start 16
03:40:55.6002 start 17
03:40:55.6159 start 18
03:40:55.6326 start 19
03:40:55.7096 start 20
03:41:05.5692 end sleep 11 diff 10015.8226
03:41:05.5692 end sleep 12 diff 10015.8226
03:41:05.5848 end sleep 13 diff 10015.8108
03:41:05.6004 end sleep 14 diff 10015.8128
03:41:05.6942 end sleep 15 diff 10015.8205
03:41:05.5848 end 12 diff 20099.3218
03:41:05.6004 end 11 diff 20130.5719
03:41:05.6161 end 13 diff 20114.9729
03:41:05.6473 end 14 diff 20130.5962
03:41:05.6942 end 15 diff 20083.7506
03:41:05.6317 start sleep 17
03:41:05.6317 start sleep 16
03:41:05.6473 start sleep 18
03:41:05.6629 start sleep 19
03:41:05.7411 start sleep 20
03:41:15.6320 end sleep 16 diff 10000.3608
03:41:15.6320 end sleep 17 diff 10000.3608
03:41:15.6477 end sleep 18 diff 10000.3727
03:41:15.6633 end sleep 19 diff 10000.3709
03:41:15.7414 end sleep 20 diff 10000.3546
03:41:15.6477 end 16 diff 20047.4375
03:41:15.6477 end 17 diff 20047.4375
03:41:15.6633 end 18 diff 20047.4295
03:41:15.7101 end 19 diff 20077.5169
03:41:15.7414 end 20 diff 20031.7909
I believe your test is a little flawed. According to this TechNet article, $PID is not an available variable in a Script Workflow.
A better test with an explanation can be found on this site
Basically, the ThrottleLimit is set to a theoretical [Int32]::MaxValue. The poster designed a better test to check parallel processing also (slightly modified):
Workflow Throttle-Me {
[cmdletbinding()]
param (
[int]$ThrottleLimit = 10,
[int]$CollectionLimit = 10
)
foreach -parallel -throttlelimit $ThrottleLimit ($n in 1..$CollectionLimit){
"Working on $n"
"{0:hh}:{0:mm}:{0:ss}" -f (Get-Date)
}
}
I can't speak for that specific cmdlet , but as you found PowerShell only uses one thread per PID. The common ways of getting around this are PSJobs and Runspaces. Runspaces are more difficult than PSJobs, but they also have significantly better performance, which makes them the popular choice. The downside is that your entire code has to resolve around them which means an entire rewrite in most cases.
Unfortunately as much as Microsoft pushes PowerShell, it is not made for performance and speed. But it does as good as it can while tied to .NET
Related
I am running a PS script to create an excel file with some text and the corresponding pictures to the text. Till now everything works as expected but when i want to choose the right column for the pictures with a counter something weird happens.
Before the if statement the counter behaves like it should, but as soon i let it output the counter inside the if statement the counter does something weird what i cannot explain.
Part of the Script
pictures is an list of file names
rows_sorted is an list of camera names
foreach ($picture in $pictures) {
foreach ($row in $rows_sorted) {
$xlcounter += 1
Write $xlcounter -> counter looks fine
if ($picture -match $row ) {
$live_picture = "C:\Snapshots\pictures\$picture"
$image = [System.Drawing.Image]::FromFile($live_picture)
$sheet | Add-ExcelImage -Image $image -Row $xlcounter -Column 4 -ResizeCell
$image.Dispose()
Write $xlcounter -> Weird things happen
}
}
}
Example output of counter outside the if statement:
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Example output of counter inside the if statement:
2 205 408 611 814 1017 1220 1423 1626 1829 2032 2235 2438 2641 2844 3047 3251 3454 3657 3860 4063
I tried to rename the counter because the same name was used in a few lines before, but that did not help either.
I have some trouble with very long subroutine with hypnotoad.
I need to run 1 minute subroutine (hardware connected requirements).
Firsly, i discover this behavior with:
my $var = 0;
Mojo::IOLoop->recurring(60 => sub {
$log->debug("starting loop: var: $var");
if ($var == 0) {
#...
#make some long (30 to 50 seconds) communication with external hardware
sleep 50; #fake reproduction of this long communication
$var = 1;
}
$log->debug("ending loop: var: $var");
})
Log:
14:13:45 2018 [debug] starting loop: var: 0
14:14:26 2018 [debug] ending loop: var: 1 #duration: 41 seconds
14:14:26 2018 [debug] starting loop: var: 0
14:15:08 2018 [debug] ending loop: var: 1 #duration: 42 seconds
14:15:08 2018 [debug] starting loop: var: 0
14:15:49 2018 [debug] ending loop: var: 1 #duration: 42 seconds
14:15:50 2018 [debug] starting loop: var: 0
14:16:31 2018 [debug] ending loop: var: 1 #duration: 41 seconds
...
3 problems:
1) Where do these 42 seconds come from? (yes, i know, 42 seconds is the answer of the universe...)
2) Why the IOLoop recuring loses his pace?
3) Why my variable is setted to 1, and just one second after, the if get a variable equal to 0?
When looped job needs 20 seconds or 25 seconds, no problem.
When looped job needs 60 secondes and used with morbo, no problem.
When looped job needs more than 40 seconds and used with hypnotoad (1 worker), this is the behavior explained here.
If i increase the "not needed" time (e.g. 120 seconds IOLoop for 60 seconds jobs, the behaviour is always the same.
It's not a problem about IOLoop, i can reproduce the same behavior outside loop.
I suspect a problem with worker killed and heart-beat, but i have no log about that.
As C# does not has the options of powershell's write-progress I start to try to 're-use' the same line:
function ReUseFails {
$x=0
$nDec = 10
while ($true) {
$x++
$a = $x.ToString().PadLeft($nDec)
$z = $x.ToString().PadLeft($nDec)
Write-Host "`r$a $z" -noNewLine
Start-Sleep -s 1
}
}
ReUseFails
I expected because of `r (carriage return) to see in the same line (the next line overrides the prev.):
1 1 # and after 1 Second in that line:
2 2 # after the 2nd second
3 3 # (and so on)
but what I get is
1 1 2 2 3 3 4 4 ...
So the carriage return r has no effect?<br>
Is there a way to 'enable' thisr??
Even when I try to use `b I see additional spots but not a back-space.
Can it be that the DOS-console can do that (to show a spinning wheel) but bot the PS-console?
Regards
Can someone explain me this short pearl code?
$batstr2 = "empty" if( $status2 & 4 );
What say the if statement ?
Already answered many times, for the case if you don't know what is the Bitwise And, here is a small example:
perl -e 'print "dec\t bin\t&4\n";printf "%d\t%8b\t%-8b\n", $_, $_, ($_ & 4) for (0..8);'
prints:
dec bin &4
0 0 0
1 1 0
2 10 0
3 11 0
4 100 100
5 101 100
6 110 100
7 111 100
8 1000 0
as you can see, when the 3rb bit from right is 1 - the $num & 4 is true.
That's using the if as a statement modifier. It's roughly the same as
if ($status & 4) {
$batstr2 = "empty";
}
and exactly the same as
($status & 4) and ($batstr2 = "empty");
a variety of constructs can be used as statement modifiers, including: if, unless, while, until, for, when. These modifiers can't be stacked (foo() if $bar for #baz won't work), you are limited for one modifer per simple statement.
That's a bitwise and - http://perldoc.perl.org/perlop.html#Bitwise-And . $status2 is being used as a bit mask and it sets $batstr2 to 'empty' if the bit is set.
It sets $batstr2 to "empty" if the 3rd least significant bit of $status2 is set - it is a logical AND mask.
I have a perl script that needs to run in the background constantly. It consists of several .pm module files and a main .pl file. What the program does is to periodically gather some data, do some computation, and finally update the result recorded in a file.
All the critical data structures are declared in the .pl file with our, and there's no package variable declared in any .pm file.
I used the function arena_table() in the Devel::Gladiator module to produce some information about the arena in the main loop, and found that the SVs of type SCALAR and GLOB are increasing slowly, resulting in a gradual increase in the memory usage.
The output of arena_table (I reformat them, omitting the title. after a long enough period, only the first two number is increasing):
2013-05-17#11:24:34 36235 3924 3661 3642 3376 2401 720 201 27 23 18 13 13 10 2 2 1 1 1 1 1 1 1 1 1 1
After running for some time:
2013-05-17#12:05:10 50702 46169 36910 4151 3995 3924 2401 720 274 201 26 23 18 13 13 10 2 2 1 1 1 1 1 1 1 1 1
The main loop is something like:
our %hash1 = ();
our %hash2 = ();
# and some more package variables ...
# all are hashes
do {
my $nowtime = time();
collect_data($nowtime);
if (calculate() == 1) {
update();
}
sleep 1;
get_mem_objects(); # calls arena_table()
} while (1);
Except get_mem_objects, other functions will operate on the global hashes declared by our. In update, the program will do some log rotation, the code is like:
sub rotate_history() {
my $i = $main::HISTORY{'count'};
if ($i == $main::CONFIG{'times'}{'total'}) {
for ($i--; $i >= 1; $i--) {
$main::HISTORY{'data'}{$i} = dclone($main::HISTORY{'data'}{$i-1});
}
} else {
for (; $i >= 1; $i--) {
$main::HISTORY{'data'}{$i} = dclone($main::HISTORY{'data'}{$i-1});
}
}
$main::HISTORY{'data'}{'0'} = dclone(\%main::COLLECT);
if ($main::HISTORY{'count'} < $main::CONFIG{'times'}{'total'}) {
$main::HISTORY{'count'}++;
}
}
If I comment the calls to this function, in the final report given by Devel::Gladiator, only the SVs of type SCALAR is increasing, the number of GLOBs will finally enter a stable state. I doubt the dclone may cause the problem here.
My questions are,
what exactly does the information given by that module mean? The statements in the perldoc is a little vague for a perl newbie like me.
And, what are the common skills to lower the memory usage of long-running perl scripts?
I know that package variables are stored in the arena, but how about the lexical variables? How are the memory consumed by them managed?